Kubernetes logs

StackState Self-hosted v5.1.x

Overview

In a Kubernetes setup, StackState functions are distributed across different pods and logs for each function are stored per pod and container. You can access recent logs using kubectl, although for long term storage it's recommended to set up log aggregation.

Kubernetes pods for logging

StackState logs are stored per pod and container. The table below shows the pod to access for logs relating to specific StackState functions. Note that actual pod names will include a number or random string suffix (for example, stackstate-receiver-5b9d79db86-h2hkz) and may also include the release name specified when StackState was deployed as a prefix.

Note that logs stored on pods will be regularly removed. For long term access to logs, it's advised that you set up log aggregation for your Kubernetes cluster.

StackState function	Logs on pod
API (including topology, charts and settings)	stackstate-api
Checks	stackstate-checks
Data indexing into Elasticsearch	stackstate-mm2es (metrics) stackstate-e2es (events) stackstate-trace2es (traces) stackstate-sts2es (events generated by StackState)
Data ingestion	stackstate-receiver
Event handlers	stackstate-view-health
Monitor	stackstate-checks
State propagation	stackstate-state
Synchronization	stackstate-sync
View health state	stackstate-view-health

You can access logs on a specific pod using the kubectl logs command.

For example:

$ kubectl logs stackstate-api-0

Access recent logs

Pod or container logs

The most recent logs can be retrieved from Kubernetes using the kubectl logs command. Check the pod that you need to monitor to retrieve a specific log.

For example:

# Snapshot of logs for all containers of <pod-name>
$ kubectl logs <pod-name> --all-containers=true

# Stream logs for all containers of <pod-name>
$ kubectl logs -f <pod-name> --all-containers=true

# Snapshot of logs for a specific container of <pod-name>
$ kubectl logs -c <container-name> <pod-name>

# Snapshot of logs for previous terminated container of <pod-name>
$ kubectl logs -p -c <container-name> <pod-name>

Synchronization logs

All synchronization logs can be found in a pod stackstate-sync-<suffix>. You can use the synchronization name to locate specific log information in a log snapshot.

For example:

# Logs of the synchronization for a specific Kubernetes cluster
$ kubectl logs stackstate-sync-0 | grep "Kubernetes - \<cluster-name\>"

# Logs of the Agent synchronization
$ kubectl logs stackstate-sync-0 | grep "Agent"

Log aggregation

For long term storage of StackState log data, it's advised that you set up log aggregation on your Kubernetes cluster. This can be done using a third party system for storage such as Elasticsearch, Splunk or Logz.io and a log shipper such as Logstash or Fluentd.

For more details of how this can be done, check:

• Shipping logs with Fluentd (fluentd.org)
• A complete overview of setting up log aggregation into Elasticsearch (bitnami.com)

See also

• kubectl command reference (kubernetes.io/docs)