Kubernetes

Search…
💠
Kubernetes
StackState Self-hosted v4.6.x
Overview
The Kubernetes integration is used to create a near real-time synchronization of topology and associated internal services from a Kubernetes cluster to StackState. This StackPack allows monitoring of the following:
Workloads
Nodes, pods, containers and services
Configmaps, secrets and volumes
Kubernetes is a StackState core integration.
Data flow
The Kubernetes integration collects topology data in an OpenShift cluster as well as metrics and events.
Data is retrieved by the deployed StackState Agents and then pushed to StackState via the Agent StackPack and the Kubernetes StackPack.
In StackState:
​Topology data is translated into components and relations.
​Tags defined in Kubernetes are added to components and relations in StackState.
​Metrics data is stored and accessible within StackState. Relevant metrics data is mapped to associated components and relations in StackState.
​Kubernetes events are available in the StackState UI Events Perspective and listed in the details pane on the right of the StackState UI.
​Objects changes events are created for every detected change to spec or metadata in Kubernetes objects
Setup
Prerequisites
The following prerequisites are required to install the Kubernetes StackPack and deploy the StackState Agent and Cluster Agent:
A Kubernetes Cluster must be up and running.
A recent version of Helm 3.
A user with permissions to create privileged pods, ClusterRoles and ClusterRoleBindings:
ClusterRole and ClusterRoleBinding are needed to grant StackState Agents permissions to access the Kubernetes API.
StackState Agents need to run in a privileged pod to be able to gather information on network connections and host information.
Supported container runtimes
From StackState Agent v2.16, the following container runtimes are supported:
containerd
CRI-O
Docker
Note that versions of StackState Agent prior to v2.16 support the Docker container runtime only.
Install
Install the Kubernetes StackPack from the StackState UI StackPacks > Integrations screen. You will need to provide the following parameters:
Kubernetes Cluster Name - A name to identify the cluster. This does not need to match the cluster name used in kubeconfig, however, that is usually a good candidate for a unique name.
If the Agent StackPack is not already installed, this will be automatically installed together with the Kubernetes StackPack. This is required to work with the StackState Agent, which will need to be deployed on each node in the Kubernetes cluster.
Deploy the StackState Agent and Cluster Agent
For the Kubernetes integration to retrieve topology, events and metrics data, you will need to have the following installed on your Kubernetes cluster:
StackState Agent V2 on each node in the cluster
StackState Cluster Agent on one node
kube-state-metrics
➡️ Deploy StackState Agents and kube-state-metrics.
To integrate with other services, a separate instance of the StackState Agent should be deployed on a standalone VM. It is not currently possible to configure a StackState Agent deployed on a Kubernetes cluster with checks that integrate with other services.
Status
To check the status of the Kubernetes integration, check that the StackState Cluster Agent (cluster-agent) pod and all of the StackState Agent (cluster-agent-agent) pods have status ready.
1
❯ kubectl get deployment,daemonset --namespace stackstate
2
​
3
NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
4
deployment.apps/stackstate-cluster-agent             1/1     1            1           5h14m
5
NAME                                                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
6
daemonset.apps/stackstate-cluster-agent-agent        10        10        10      10           10          <none>          5h14m
Copied!
Integration details
Data retrieved
The Kubernetes integration retrieves the following data:
​Events​
​Metrics​
​Tags​
​Topology​
Events
All Kubernetes events are retrieved from the Kubernetes cluster.
StackState Element Properties Change events will be generated for changes detected in a Kubernetes object.
Kubernetes events
The table below shows which event category will be assigned to each event type in StackState:
StackState event category
Kubernetes events
Activities
BackOff ContainerGCFailed ExceededGracePeriod FileSystemResizeSuccessful ImageGCFailed Killing NodeAllocatableEnforced NodeNotReady NodeSchedulable Preempting Pulling Pulled Rebooted Scheduled Starting Started SuccessfulAttachVolume SuccessfulDetachVolume SuccessfulMountVolume SuccessfulUnMountVolume VolumeResizeSuccessful
Alerts
NotTriggerScaleUp
Changes
Created (created container) NodeReady SandboxChanged SuccesfulCreate
Others
All other events
Object change events
The Kubernetes integration will detect changes in Kubernetes objects and will create an event of type Element Properties Change with a diff with a YAML representation of the changed object.
Element Properties Change event
Changes will be detected in the following object types:
ConfigMap
CronJob
DaemonSet
Deployment
Ingress
Job
Namespace
Node
PersistentVolume
Pod
ReplicaSet
Secret (a hash of the content will be compared)
Service
StatefulSet
Note that, in order to reduce noise of changes, the following object properties will not be compared:
metadata
managedFields
resourceVersion
annotations
kubectl.kubernetes.io/last-applied-configuration
status (except for Node, Pod and PersistentVolume objects)
You can also see the current or past YAML definition of the object in the Component properties:
Kubernetes Component properties
Metrics
The Kubernetes integration makes all metrics from the Kubernetes cluster available in StackState. Relevant metrics are automatically mapped to the associated components.
All retrieved metrics can be browsed or added to a component as a telemetry stream. Select the data source StackState Metrics and type kubernetes in the Select box to get a full list of all available metrics.
Add a Kubernetes metrics stream to a component
Topology
The Kubernetes integration retrieves components and relations for the Kubernetes cluster.
StackState Agent versions prior to 2.16: Topology information is only gathered from Kubernetes clusters that use the Docker container runtime.
Components
The following Kubernetes topology data is available in StackState as components:
Cluster
Ingress
Namespace
Job
Node
Persistent Volume
Pod
ReplicaSet
Container
Secret
ConfigMap
Service
CronJob
StatefulSet
DaemonSet
Volume
Deployment
​
Relations
The following relations between components are retrieved:
Container → Volume
CronJob → Job
DaemonSet → Pod
Deployment → ReplicaSet
Job → Pod
Ingress → Service
Namespace → CronJob, DaemonSet, Deployment, Job, ReplicaSet, Service, StatefulSet
Node → Cluster relation
Pod → ConfigMap, Container, Deployment, Node, Secret, Volume
ReplicaSet → Pod
Service → ExternalService, Pod
StatefulSet → Pod
Direct communication between processes
Process → Process communication via Kubernetes service
Process → Process communication via headless Kubernetes service
Traces
The Kubernetes integration does not retrieve any traces data.
Tags
All tags defined in Kubernetes will be retrieved and added to the associated components and relations in StackState.
REST API endpoints
The StackState Agent talks to the kubelet and kube-state-metrics API.
The StackState Agent and Cluster Agent connect to the Kubernetes API to retrieve cluster wide information and Kubernetes events. The following API endpoints used:
Resource type
REST API endpoint
Cluster > ComponentStatus
GET /api/v1/componentstatuses
Cluster > Event
GET /apis/events.k8s.io/v1/events
Cluster > Namespace
GET /api/v1/namespaces
Cluster > Node
GET /api/v1/nodes
Services > Endpoints
GET /api/v1/namespaces/{namespace}/endpoints
Services > Ingress
GET /apis/networking.k8s.io/v1/namespaces/{namespace}/ingresses
Services > Service
GET /api/v1/namespaces/{namespace}/services
Workloads > CronJob
GET /apis/batch/v1beta1/namespaces/{namespace}/cronjobs
Workloads > DaemonSet
GET /apis/apps/v1/namespaces/{namespace}/daemonsets
Workloads > Deployment
GET /apis/apps/v1/namespaces/{namespace}/deployments
Workloads > Job
GET /apis/batch/v1/namespaces/{namespace}/jobs
Workloads > Pod
GET /api/v1/namespaces/{namespace}/pods
Workloads > ReplicaSet
GET /apis/apps/v1/namespaces/{namespace}/replicasets
Workloads > StatefulSet
GET /apis/apps/v1/namespaces/{namespace}/statefulsets
Config and Storage > ConfigMap
GET /api/v1/namespaces/{namespace}/configmaps
Config and Storage > Secret
GET /api/v1/secrets
Config and Storage > PersistentVolume
GET /api/v1/persistentvolumes
Config and Storage > PersistentVolumeClaimSpec
GET /api/v1/namespaces/{namespace}/persistentvolumeclaims
Config and Storage > VolumeAttachment
GET /apis/storage.k8s.io/v1/volumeattachments
​
/version
For further details, refer to the Kubernetes API documentation (kubernetes.io).
Component actions
A number of actions are added to StackState when the Kubernetes StackPack is installed. They are available from the Actions section on the right of the screen when a Kubernetes component is selected or from the component context menu, displayed when you hover over a Kubernetes component in the Topology Perspective
Action
Available for component types
Description
Show configuration and storage
pods containers
Display the selected pod or container with its configmaps, secrets and volumes
Show dependencies (deep)
deployment replicaset replicationcontroller statefulset daemonset job cronjob pod
Displays all dependencies (up to 6 levels deep) of the selected pod or workload.
Show pods
deployment replicaset replicationcontroller statefulset daemonset job cronjob
Displays the pods for the selected workload.
Show pods & services
namespace
Opens a view for the pods/services in the selected namespace
Show services
namespace
Open a view for the service and ingress components in the selected namespace
Show workloads
namespace
Show workloads in the selected namespace
Details of installed actions can be found in the StackState UI Settings > Actions > Component Actions screen.
Kubernetes views in StackState
When the Kubernetes integration is enabled, the following Kubernetes views are available in StackState for each cluster:
Kubernetes - Applications -
Kubernetes - Infrastructure -
Kubernetes - Namespaces -
Kubernetes - Workload Controllers -
Open source
The code for the StackState Agent Kubernetes check is open source and available on GitHub at:
​https://github.com/StackVista/stackstate-agent/tree/master/pkg/collector/corechecks/cluster​
​https://github.com/stackvista/stackstate-agent​
​https://github.com/StackVista/stackstate-agent-integrations/tree/master/kubernetes​
Troubleshooting
Troubleshooting steps for any known issues can be found in the StackState support Knowledge base.
Uninstall
To uninstall the Kubernetes StackPack, go to the StackState UI StackPacks > Integrations > Kubernetes screen and click UNINSTALL. All Kubernetes StackPack specific configuration will be removed from StackState.
Release notes
Kubernetes StackPack v3.9.9 (2022-03-02)
Improvement: Documentation for agent.containerRuntime.customSocketPath option.
Kubernetes StackPack v3.9.8 (2021-11-30)
Bug Fix: Support nodes without instanceId
Kubernetes StackPack v3.9.7 (2021-10-06)
Bug Fix: Fix metrics for generic events
Kubernetes StackPack v3.9.6 (2021-08-20)
Improvement: Add description to Views
Kubernetes StackPack v3.9.5 (2021-07-14)
Improvement: Documentation update
Improvement: Update of stackstate.url for the installation documentation of the StackState Agent
See also
​Deploy StackState Agent V2, the Cluster Agent and kube-state-metrics​
​StackState Agent V2 StackPack​
​StackState Agent Kubernetes check (github.com)​
​StackState Cluster Agent Helm Chart (github.com)​
​Kubernetes API documentation (kubernetes.io)​