StackState for Kubernetes troubleshooting Self-hosted
All of StackState's own components can run without any extra permissions. However, to install StackState successfully, you need some additional privileges, or ensure that the requirements described in this page are met.
To allow communication between StackState services StackState uses Kubernetes service accounts. To be able to verify their validity and roles the helm chart creates a
ClusterRoleBindingresources. Creating these cluster-wide resources is often prohibited for users that aren't a Kubernetes/OpenShift administrator. For that case the creation can be disabled and instead the roles and role bindings need to be created manually by your cluster admin.
The automatic creation of cluster-wide resources during installation of StackState can be disabled by adding the following section to the
values.yamlfile used to install StackState:
If you need to manually create the cluster-wide resources, ask your Kubernetes/OpenShift administrator to create the 3 resources below in the clsuter.
Verify that you specify the correct service account and namespace for the bound
ServiceAccountfor both of the
ClusterRoleBindingresources. The example assumes the
stackstatenamespace is used, if some other namespace is used changed the namespace in the examples. Also the service accounts referenced need to be changed to
- kind: ServiceAccount
- kind: ServiceAccount
StackState uses Elasticsearch to store its indices. There are some additional requirements for the nodes that Elasticsearch runs on.
vm.max_map_countLinux system setting is usually lower than required for Elasticsearch to start, an init container is used that runs in privileged mode and as the root user. The init container is enabled by default to allow the
vm.max_map_countsystem setting to be changed.
In case you or your Kubernetes/OpenShift administrators don't want the privileged Elasticsearch init container to be enabled by default, you can disable this behavior in the file
values.yamlused to install StackState:
If this is disabled, you will need to ensure that the
vm.max_map_countsetting is changed from its common default value of
262144. If this isn't done, Elasticsearch will fail to start up and its pods will be in a restart loop.
To inspect the current
vm.max_map_countsetting, run the following command. Note that it runs a privileged pod:
kubectl run -i --tty sysctl-check-max-map-count --privileged=true --image=busybox --restart=Never --rm=true -- sysctl vm.max_map_count
If the current
vm.max_map_countsetting isn't at least
262144, it will need to be increased in a different way or Elasticsearch will fail to start up and its pods will be in a restart loop. The logs will contain an error message like this:
bootstrap checks failed
max virtual memory areas vm.max_map_count  is too low, increase to at least 
Depending on what your Kubernetes/OpenShift administrators prefer, the
vm.max_map_countcan be set to a higher default on all nodes by either changing the default node configuration (for example via init scripts) or by having a DaemonSet do this right after node startup. The former is very dependent on your clsuter setup, so there are no general solutions there.
Below is an example that can be used as a starting point for a DaemonSet to change the
# Make sure the setting always gets changed as soon as possible:
- effect: NoSchedule
- effect: NoExecute
# Optional node selector (assumes nodes for Elasticsearch are labeled `elasticsearch:yes`
# elasticsearch: yes
- name: set-vm-max-map-count
command: ["sysctl", "-w", "vm.max_map_count=262144"]
# A pause container is needed to prevent a restart loop of the pods in the daemonset
# See also this Kubernetes issue https://github.com/kubernetes/kubernetes/issues/36601
- name: pause
To limit the number of nodes that this is applied to, nodes can be labeled. NodeSelectors on both this DaemonSet, as shown in the example, and the Elasticsearch deployment can then be set to run only on nodes with the specific label. For Elasticsearch, the node selector can be specified via the values: