Kubernetes
StackState SaaS
The Kubernetes integration is used to create a near real-time synchronization of topology and associated internal services from a Kubernetes cluster to StackState. This StackPack allows monitoring of the following:
- Workloads
- Nodes, pods, containers and services
- Configmaps, secrets and volumes
Data flow
The Kubernetes integration collects topology data in a Kubernetes cluster as well as metrics and events.
- Data is retrieved by the deployed StackState Agents and then pushed to StackState via the Agent StackPack and the Kubernetes StackPack.
- In StackState:
- Metrics data is stored and accessible within StackState. Relevant metrics data is mapped to associated components and relations in StackState.
- Kubernetes events are available in the StackState UI Events Perspective. They are also included in the Event list in the right panel View summary tab and the details tabs - Component details and Direct relation details.
- Object change events are created for every detected change to
spec
ormetadata
in Kubernetes objects
The following prerequisites are required to install the Kubernetes StackPack and deploy the StackState Agent and Cluster Agent:
- A Kubernetes Cluster must be up and running.
- A recent version of Helm 3.
- A user with permissions to create privileged pods, ClusterRoles and ClusterRoleBindings:
- ClusterRole and ClusterRoleBinding are needed to grant StackState Agents permissions to access the Kubernetes API.
- StackState Agents need to run in a privileged pod to be able to gather information on network connections and host information.
From StackState Agent V2.16, the following container runtimes are supported:
- containerd
- CRI-O
- Docker
Note that versions of StackState Agent prior to v2.16 support the Docker container runtime only.
Install the Kubernetes StackPack from the StackState UI StackPacks > Integrations screen. You will need to provide the following parameters:
- Kubernetes Cluster Name - A name to identify the cluster. This does not need to match the cluster name used in
kubeconfig
, however, that is usually a good candidate for a unique name.
If the Agent StackPack is not already installed, this will be automatically installed together with the Kubernetes StackPack. This is required to work with the StackState Agent, which will need to be deployed on each node in the Kubernetes cluster.
For the Kubernetes integration to retrieve topology, events and metrics data, you will need to have the following installed on your Kubernetes cluster:
- StackState Agent V2 on each node in the cluster
- StackState Cluster Agent on one node
- StackState Checks Agent on one node
- kube-state-metrics
To integrate with other services, a separate instance of the StackState Agent should be deployed on a standalone VM. It is not currently possible to configure a StackState Agent deployed on a Kubernetes cluster with checks that integrate with other services.
The kubernetes_state check is responsible for gathering metrics from kube-state-metrics and sending them to StackState. The kubernetes_state check runs in the StackState Checks Agent by default and is configured in the StackState Cluster Agent.
The default URL that the kubernetes_state check uses is:
http://<release-name>-kube-state-metrics.<namespace>.svc.cluster.local:8080/metrics
If an alternative kube-state-metrics pod (i.e. Prometheus) is installed, the default StackState kube-state-metrics pod can be disabled and the kubernetes_state check redirected to the alternative service:
- 1.Update the
values.yaml
file used to deploy thechecks-agent
, for example:dependencies:kubeStateMetrics:enabled: falsechecksAgent:kubeStateMetrics:url: http://YOUR_KUBE_STATE_METRICS_SERVICE_NAME:8080/metrics - 2.Deploy the
cluster_agent
using the updatedvalues.yaml
:helm upgrade --install \--namespace stackstate \--create-namespace \--set-string 'stackstate.apiKey'='<STACKSTATE_RECEIVER_API_KEY>' \--set-string 'stackstate.cluster.name'='<KUBERNETES_CLUSTER_NAME>' \--set-string 'stackstate.cluster.authToken=<CLUSTER_AUTH_TOKEN>' \--set-string 'stackstate.url'='<STACKSTATE_RECEIVER_API_ADDRESS>' \--values values.yaml \stackstate-agent stackstate/stackstate-agent
To check the status of the Kubernetes integration, check that the StackState Cluster Agent (
cluster-agent
) pod, StackState Checks Agent pod (checks-agent
) and all of the StackState Agent (node-agent
) pods have status ready.❯ kubectl get deployment,daemonset --namespace stackstate
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/stackstate-agent-cluster-agent 1/1 1 1 5h14m
deployment.apps/stackstate-agent-checks-agent 1/1 1 1 5h14m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/stackstate-agent-node-agent 10 10 10 10 10 <none> 5h14m
The Kubernetes integration retrieves the following data:
- StackState
Element Properties Change
events will be generated for changes detected in a Kubernetes object.
Kubernetes events
The Kubernetes integration retrieves all events from the Kubernetes cluster. The table below shows which event category will be assigned to each event type in StackState:
StackState event category | Kubernetes events |
---|---|
Activities | BackOff ContainerGCFailed ExceededGracePeriod FileSystemResizeSuccessful ImageGCFailed Killing NodeAllocatableEnforced NodeNotReady NodeSchedulable Preempting Pulling Pulled Rebooted Scheduled Starting Started SuccessfulAttachVolume SuccessfulDetachVolume SuccessfulMountVolume SuccessfulUnMountVolume VolumeResizeSuccessful |
Alerts | NotTriggerScaleUp |
Changes | Created (created container) NodeReady SandboxChanged SuccessfulCreate SuccessfulDelete Completed |
Others | All other events |
Object change events
The Kubernetes integration will detect changes in Kubernetes objects and will create an event of type
Element Properties Change
with a diff with a YAML representation of the changed object.
Element Properties Change event
Changes will be detected in the following object types:
ConfigMap
CronJob
DaemonSet
Deployment
Ingress
Job
Namespace
Node
PersistentVolume
Pod
ReplicaSet
Secret
(a hash of the content will be compared)Service
StatefulSet
Note that, in order to reduce noise of changes, the following object properties will not be compared:
metadata
managedFields
resourceVersion
annotations
kubectl.kubernetes.io/last-applied-configuration
status
(except forNode
,Pod
andPersistentVolume
objects)

Kubernetes Component properties
The Kubernetes integration makes all metrics from the Kubernetes cluster available in StackState. Relevant metrics are automatically mapped to the associated components.
All retrieved metrics can be browsed or added to a component as a telemetry stream. Select the data source StackState Metrics and type
kubernetes
in the Select box to get a full list of all available metrics.
Add a Kubernetes metrics stream to a component
The Kubernetes integration retrieves components and relations for the Kubernetes cluster.
StackState Agent versions prior to 2.16: Topology information is only gathered from Kubernetes clusters that use the Docker container runtime.
Components
The following Kubernetes topology data is available in StackState as components:
Cluster | Ingress |
Namespace | Job |
Node | Persistent Volume |
Pod | ReplicaSet |
Container | Secret |
ConfigMap | Service |
CronJob | StatefulSet |
DaemonSet | Volume |
Deployment | |
Relations
The following relations between components are retrieved:
- Container → PersistentVolume, Volume
- CronJob → Job
- DaemonSet → Pod
- Deployment → ReplicaSet
- Job → Pod
- Ingress → Service
- Namespace → CronJob, DaemonSet, Deployment, Job, ReplicaSet, Service, StatefulSet
- Node → Cluster relation
- Pod → ConfigMap, Container, Deployment, Node, Secret, Volume
- ReplicaSet → Pod
- Service → ExternalService, Pod
- StatefulSet → Pod
- Direct communication between processes
- Process → Process communication via Kubernetes service
- Process → Process communication via headless Kubernetes service
The Kubernetes integration does not retrieve any traces data.
All tags defined in Kubernetes will be retrieved and added to the associated components and relations in StackState as labels.
The following labels will also be added to imported elements as relevant:
image_name
kube_cluster_name
kube_container_name
kube_cronjob
kube_daemon_set
kube_deployment
kube_job
kube_namespace
kube_replica_set
kube_replication_controller
kube_stateful_set
pod_name
pod_phase
The StackState Agent talks to the kubelet and kube-state-metrics API.
The StackState Agent and Cluster Agent connect to the Kubernetes API to retrieve cluster wide information and Kubernetes events. The following API endpoints used:
Resource type | REST API endpoint |
---|---|
Cluster > ComponentStatus | GET /api/v1/componentstatuses |
Cluster > Event | GET /apis/events.k8s.io/v1/events |
Cluster > Namespace | GET /api/v1/namespaces |
Cluster > Node | GET /api/v1/nodes |
Services > Endpoints | GET /api/v1/namespaces/{namespace}/endpoints |
Services > Ingress | GET /apis/networking.k8s.io/v1/namespaces/{namespace}/ingresses |
Services > Service | GET /api/v1/namespaces/{namespace}/services |
Workloads > CronJob | GET /apis/batch/v1beta1/namespaces/{namespace}/cronjobs |
Workloads > DaemonSet | GET /apis/apps/v1/namespaces/{namespace}/daemonsets |
Workloads > Deployment | GET /apis/apps/v1/namespaces/{namespace}/deployments |
Workloads > Job | GET /apis/batch/v1/namespaces/{namespace}/jobs |
Workloads > Pod | GET /api/v1/namespaces/{namespace}/pods |
Workloads > ReplicaSet | GET /apis/apps/v1/namespaces/{namespace}/replicasets |
Workloads > StatefulSet | GET /apis/apps/v1/namespaces/{namespace}/statefulsets |
Config and Storage > ConfigMap | GET /api/v1/namespaces/{namespace}/configmaps |
Config and Storage > Secret | GET /api/v1/secrets |
Config and Storage > PersistentVolume | GET /api/v1/persistentvolumes |
Config and Storage > PersistentVolumeClaimSpec | GET /api/v1/namespaces/{namespace}/persistentvolumeclaims |
Config and Storage > VolumeAttachment | GET /apis/storage.k8s.io/v1/volumeattachments |
| /version |
| /healthz |
A number of actions are added to StackState when the Kubernetes StackPack is installed. They are available from the Actions section in the right panel details tab - Component details - when a Kubernetes component is selected or from the component context menu, displayed when you hover the mouse pointer over a Kubernetes component in the Topology Perspective
Action | Available for component types | Description |
---|---|---|
Show configuration and storage | pods containers | Display the selected pod or container with its configmaps, secrets and volumes |
Show dependencies (deep) | deployment replicaset replicationcontroller statefulset daemonset job cronjob pod | Displays all dependencies (up to 6 levels deep) of the selected pod or workload. |
Show pods | deployment replicaset replicationcontroller statefulset daemonset job cronjob | Displays the pods for the selected workload. |
Show pods and services | namespace | Opens a view for the pods/services in the selected namespace |
Show services | namespace | Open a view for the service and ingress components in the selected namespace |
Show workloads | namespace | Show workloads in the selected namespace |
Details of installed actions can be found in the StackState UI Settings > Actions > Component Actions screen.
When the Kubernetes integration is enabled, the following Kubernetes views are available in StackState for each cluster:
- Kubernetes - Applications -
- Kubernetes - Infrastructure -
- Kubernetes - Namespaces -
- Kubernetes - Workload Controllers -
The code for the StackState Agent Kubernetes check is open source and available on GitHub at:
To uninstall the Kubernetes StackPack, go to the StackState UI StackPacks > Integrations > Kubernetes screen and click UNINSTALL. All Kubernetes StackPack specific configuration will be removed from StackState.
See the Kubernetes Agent documentation for instructions on how to uninstall the StackState Cluster Agent and the StackState Agent from your Kubernetes cluster.
Kubernetes StackPack v3.10.2 (2022-09-28)
- Improvement: Updated documentation.
Kubernetes StackPack v3.10.1 (2022-09-15)
- Improvement: Added validation for cluster name
Kubernetes StackPack v3.9.13 (2022-06-21)
- Bug Fix: Fixed description for services/ingresses view.
Kubernetes StackPack v3.9.12 (2022-06-03)
- Improvement: Updated documentation.
Kubernetes StackPack v3.9.11 (2022-05-23)
- Bug Fix: Fixed broken link in integration StackState Agent V2 integration documentation.
Kubernetes StackPack v3.9.10 (2022-04-11)
- Bug Fix: Show kubernetes view names on StackPack instance
Kubernetes StackPack v3.9.9 (2022-03-02)
- Improvement: Documentation for
agent.containerRuntime.customSocketPath
option.
Kubernetes StackPack v3.9.8 (2021-11-30)
- Bug Fix: Support nodes without instanceId
Last modified 11mo ago