Kubernetes logs
StackState Self-hosted v5.1.x
In a Kubernetes setup, StackState functions are distributed across different pods and logs for each function are stored per pod and container. You can access recent logs using
kubectl, although for long term storage it's recommended to set up log aggregation.StackState logs are stored per pod and container. The table below shows the pod to access for logs relating to specific StackState functions. Note that actual pod names will include a number or random string suffix (for example,
stackstate-receiver-5b9d79db86-h2hkz) and may also include the release name specified when StackState was deployed as a prefix.Note that logs stored on pods will be regularly removed. For long term access to logs, it's advised that you set up log aggregation for your Kubernetes cluster.
StackState function | Logs on pod |
|---|---|
API (including topology, charts and settings) | stackstate-api |
Checks | stackstate-checks |
Data indexing into Elasticsearch | stackstate-mm2es (metrics) stackstate-e2es (events) stackstate-trace2es (traces) stackstate-sts2es (events generated by StackState) |
Data ingestion | stackstate-receiver |
Event handlers | stackstate-view-health |
Monitor | stackstate-checks |
State propagation | stackstate-state |
Synchronization | stackstate-sync |
View health state | stackstate-view-health |
You can access logs on a specific pod using the
kubectl logs command.For example:
$ kubectl logs stackstate-api-0
The most recent logs can be retrieved from Kubernetes using the
kubectl logs command. Check the pod that you need to monitor to retrieve a specific log.For example:
# Snapshot of logs for all containers of <pod-name>
$ kubectl logs <pod-name> --all-containers=true
# Stream logs for all containers of <pod-name>
$ kubectl logs -f <pod-name> --all-containers=true
# Snapshot of logs for a specific container of <pod-name>
$ kubectl logs -c <container-name> <pod-name>
# Snapshot of logs for previous terminated container of <pod-name>
$ kubectl logs -p -c <container-name> <pod-name>
All synchronization logs can be found in a pod
stackstate-sync-<suffix>. You can use the synchronization name to locate specific log information in a log snapshot.For example:
# Logs of the synchronization for a specific Kubernetes cluster
$ kubectl logs stackstate-sync-0 | grep "Kubernetes - \<cluster-name\>"
# Logs of the Agent synchronization
$ kubectl logs stackstate-sync-0 | grep "Agent"
For long term storage of StackState log data, it's advised that you set up log aggregation on your Kubernetes cluster. This can be done using a third party system for storage such as Elasticsearch, Splunk or Logz.io and a log shipper such as Logstash or Fluentd.
For more details of how this can be done, check:
Last modified 1mo ago