StackState log files

Work with the StackState log files on Kubernetes and Linux

This page describes StackState version 4.1.

The StackState 4.1 version range is End of Life (EOL) and no longer supported. We encourage customers still running the 4.1 version range to upgrade to a more recent release.

Go to the documentation for the latest StackState release.

• Kubernetes setup

• Linux setup

Kubernetes

In a Kubernetes setup, StackState functions are distributed across different pods and logs for each function are stored per pod and container. You can access recent logs using kubectl, although for long term storage it is recommended to set up log aggregation.

Access recent logs

The most recent logs can be retrieved from Kubernetes using the kubectl logs command. For example:

# Snapshot of logs for all containers of <pod-name>
kubectl logs <pod-name> --all-containers=true

# Stream logs for all containers of <pod-name>
kubectl logs -f <pod-name> --all-containers=true

# Snapshot of logs for a specific container of <pod-name>
kubectl logs -c <container-name> <pod-name>

# Snapshot of logs for previous terminated container of <pod-name>
kubectl logs -p -c <container-name> <pod-name>

You can use the synchronization name to locate specific log information in a log snapshot. For example:

# Logs of the synchronization for a specific Kubernetes cluster
kubectl logs stackstate-server-0 | grep "Kubernetes - \<cluster-name\>"

# Logs of the agent synchronization
kubectl logs stackstate-server-0 | grep "Agent"

Find StackState logs on Kubernetes pods

StackState logs are stored per pod and container. The table below shows the pod to access for logs relating to specific StackState functions. Note that actual pod names will include a number or random string suffix (e.g. stackstate-receiver-5b9d79db86-h2hkz) and may also include the release name specified when StackState was deployed as a prefix.

StackState function	Pod
API (including topology, charts and settings)	stackstate-server
Checks	stackstate-server
Data indexing into Elasticsearch	stackstate-mm2es (metrics) stackstate-e2es (events) stackstate-trace2es (traces) stackstate-sts2es (events generated by StackState)
Data ingestion	stackstate-receiver
Event handlers	stackstate-server
State propagation	stackstate-server
Synchronization	stackstate-server
View health state	stackstate-server

You can access logs on a specific pod using the kubectl logs command. For example:

kubectl logs stackstate-server-0

Note that logs stored on pods will be regularly removed. For long term access to logs, it is advised that you set up log aggregation for your Kubernetes cluster.

Log aggregation

For long term storage of StackState log data, it is advised that you set up log aggregation on your Kubernetes cluster. This can be done using a third party system for storage such as Elasticsearch, Splunk or Logz.io and a log shipper such as Logstash or Fluentd.

For more details of how this can be done, check:

• Shipping logs with Fluentd (fluentd.org)

• A complete overview of setting up log aggregation into Elasticsearch (bitnami.com)

Linux

In a Linux setup, StackState keeps all log files in the var/log subdirectory of the StackState installation directory. By default this is /opt/stackstate/var/log. In case of a two-node installation, logs are kept in the var/log directory on each node. Note that the logs are node-specific - the StackState node keeps StackState related logs and the StackGraph node keeps logs related to StackGraph.

var/log details

Inside var/log, there is a subdirectory for each type of logs kept by StackState. By default, each subdirectory has a size cap of 2 GB; when exceeded, the oldest files are deleted. Example of var/log contents:

.:
drwxr-xr-x. 2 stackstate stackstate     4096 Dec 18 00:00 correlate
drwxr-xr-x. 2 stackstate stackstate     4096 Oct 23 00:00 elasticsearch
drwxr-xr-x. 2 stackstate stackstate    12288 Dec 18 00:00 elasticsearch7
drwxr-xr-x. 2 stackstate stackstate     4096 Dec 18 00:00 kafka
drwxr-xr-x. 2 stackstate stackstate     4096 Dec 18 00:20 kafka-to-es
drwxr-xr-x. 2 stackstate stackstate     4096 Nov 14 12:35 license-check
-rw-r--r--. 1 stackstate stackstate     2987 Dec 17 13:43 processmanager.2019-12-16.0.log
-rw-r--r--. 1 stackstate stackstate    65240 Dec 17 23:55 processmanager.2019-12-17.0.log
-rw-r--r--. 1 stackstate stackstate    10656 Dec 18 12:03 processmanager.log
drwxr-xr-x. 2 stackstate stackstate     4096 Oct 15 15:25 stackgraph
drwxr-xr-x. 2 stackstate stackstate     4096 Dec 16 09:03 stackpack
drwxr-xr-x. 2 stackstate stackstate     4096 Dec 17 13:21 stackstate
-rw-r--r--. 1 stackstate stackstate  6136841 Dec 16 23:59 stackstate.2019-12-16.11.log
-rw-r--r--. 1 stackstate stackstate 52429560 Dec 17 12:02 stackstate.2019-12-17.0.log
-rw-r--r--. 1 stackstate stackstate 52429586 Dec 17 20:39 stackstate.2019-12-17.1.log
-rw-r--r--. 1 stackstate stackstate 20252875 Dec 17 23:59 stackstate.2019-12-17.2.log
-rw-r--r--. 1 stackstate stackstate 52449712 Dec 18 00:37 stackstate.2019-12-18.0.log
-rw-r--r--. 1 stackstate stackstate 52441741 Dec 18 06:22 stackstate.2019-12-18.10.log
-rw-r--r--. 1 stackstate stackstate 52450906 Dec 18 07:27 stackstate.2019-12-18.11.log
-rw-r--r--. 1 stackstate stackstate 52478100 Dec 18 08:33 stackstate.2019-12-18.12.log
-rw-r--r--. 1 stackstate stackstate 52490055 Dec 18 09:37 stackstate.2019-12-18.13.log
-rw-r--r--. 1 stackstate stackstate 52428811 Dec 18 10:42 stackstate.2019-12-18.14.log
-rw-r--r--. 1 stackstate stackstate 52496521 Dec 18 11:48 stackstate.2019-12-18.15.log
-rw-r--r--. 1 stackstate stackstate 52445570 Dec 18 12:54 stackstate.2019-12-18.16.log
-rw-r--r--. 1 stackstate stackstate 52435555 Dec 18 01:10 stackstate.2019-12-18.1.log
-rw-r--r--. 1 stackstate stackstate 52453947 Dec 18 01:44 stackstate.2019-12-18.2.log
-rw-r--r--. 1 stackstate stackstate 52449896 Dec 18 02:19 stackstate.2019-12-18.3.log
-rw-r--r--. 1 stackstate stackstate 52429347 Dec 18 02:54 stackstate.2019-12-18.4.log
-rw-r--r--. 1 stackstate stackstate 52435159 Dec 18 03:29 stackstate.2019-12-18.5.log
-rw-r--r--. 1 stackstate stackstate 52486191 Dec 18 04:03 stackstate.2019-12-18.6.log
-rw-r--r--. 1 stackstate stackstate 52440210 Dec 18 04:38 stackstate.2019-12-18.7.log
-rw-r--r--. 1 stackstate stackstate 52436912 Dec 18 05:13 stackstate.2019-12-18.8.log
-rw-r--r--. 1 stackstate stackstate 52432884 Dec 18 05:47 stackstate.2019-12-18.9.log
-rw-r--r--. 1 stackstate stackstate   469460 Dec 18 12:54 stackstate.log
drwxr-xr-x. 2 stackstate stackstate     4096 Dec 18 00:00 stackstate-receiver
-rw-r--r--. 1 stackstate stackstate  1067419 Nov 20 10:04 stackstate-service.log
drwxr-xr-x. 2 stackstate stackstate     4096 Dec 18 12:49 sync
-rw-r--r--. 1 stackstate stackstate 52437278 Dec 18 12:34 transaction.metrics.2019-12-18.100.log
-rw-r--r--. 1 stackstate stackstate 52474749 Dec 18 12:41 transaction.metrics.2019-12-18.101.log
-rw-r--r--. 1 stackstate stackstate 52471248 Dec 18 12:49 transaction.metrics.2019-12-18.102.log
-rw-r--r--. 1 stackstate stackstate 52432268 Dec 18 10:22 transaction.metrics.2019-12-18.83.log
-rw-r--r--. 1 stackstate stackstate 52455578 Dec 18 10:30 transaction.metrics.2019-12-18.84.log
-rw-r--r--. 1 stackstate stackstate 52435647 Dec 18 10:39 transaction.metrics.2019-12-18.85.log
-rw-r--r--. 1 stackstate stackstate 52548076 Dec 18 10:49 transaction.metrics.2019-12-18.86.log
-rw-r--r--. 1 stackstate stackstate 52434274 Dec 18 10:57 transaction.metrics.2019-12-18.87.log
-rw-r--r--. 1 stackstate stackstate 52537926 Dec 18 11:05 transaction.metrics.2019-12-18.88.log
-rw-r--r--. 1 stackstate stackstate 52460399 Dec 18 11:12 transaction.metrics.2019-12-18.89.log
-rw-r--r--. 1 stackstate stackstate 52455265 Dec 18 11:19 transaction.metrics.2019-12-18.90.log
-rw-r--r--. 1 stackstate stackstate 52452737 Dec 18 11:26 transaction.metrics.2019-12-18.91.log
-rw-r--r--. 1 stackstate stackstate 52450217 Dec 18 11:34 transaction.metrics.2019-12-18.92.log
-rw-r--r--. 1 stackstate stackstate 52430615 Dec 18 11:41 transaction.metrics.2019-12-18.93.log
-rw-r--r--. 1 stackstate stackstate 52466370 Dec 18 11:48 transaction.metrics.2019-12-18.94.log
-rw-r--r--. 1 stackstate stackstate 52449864 Dec 18 11:55 transaction.metrics.2019-12-18.95.log
-rw-r--r--. 1 stackstate stackstate 52430948 Dec 18 12:03 transaction.metrics.2019-12-18.96.log
-rw-r--r--. 1 stackstate stackstate 52442265 Dec 18 12:11 transaction.metrics.2019-12-18.97.log
-rw-r--r--. 1 stackstate stackstate 52502949 Dec 18 12:19 transaction.metrics.2019-12-18.98.log
-rw-r--r--. 1 stackstate stackstate 52479825 Dec 18 12:26 transaction.metrics.2019-12-18.99.log
-rw-r--r--. 1 stackstate stackstate 44666747 Dec 18 12:54 transaction.metrics.log

./correlate:
-rw-r--r--. 1 stackstate stackstate 10389822 Dec 18 12:54 correlate.log
-rw-r--r--. 1 stackstate stackstate  3913232 Dec 18 12:54 worker.correlateConnections-1.log
-rw-r--r--. 1 stackstate stackstate  2629015 Dec 18 12:54 worker.correlateConnections-2.log
-rw-r--r--. 1 stackstate stackstate  3731346 Dec 18 12:54 worker.correlateConnections-3.log

./elasticsearch:
-rw-r--r--. 1 stackstate stackstate   3401 Oct 15 15:46 stackstate-2019-10-15.log
-rw-r--r--. 1 stackstate stackstate   3484 Oct 16 08:27 stackstate-2019-10-16.log
-rw-r--r--. 1 stackstate stackstate  50560 Oct 17 08:24 stackstate-2019-10-17.log
-rw-r--r--. 1 stackstate stackstate   3678 Oct 18 07:01 stackstate-2019-10-18.log
-rw-r--r--. 1 stackstate stackstate  32024 Oct 21 21:05 stackstate-2019-10-21.log
-rw-r--r--. 1 stackstate stackstate 129371 Oct 22 23:19 stackstate-2019-10-22.log
-rw-r--r--. 1 stackstate stackstate      0 Oct 15 15:46 stackstate_deprecation.log
-rw-r--r--. 1 stackstate stackstate      0 Oct 15 15:46 stackstate_index_indexing_slowlog.log
-rw-r--r--. 1 stackstate stackstate      0 Oct 15 15:46 stackstate_index_search_slowlog.log
-rw-r--r--. 1 stackstate stackstate   4664 Oct 23 10:29 stackstate.log

./elasticsearch7:
-rw-r--r--. 1 stackstate stackstate    75279 Dec 18 00:00 stackstate-2019-12-17-1.json.gz
-rw-r--r--. 1 stackstate stackstate    70897 Dec 18 00:00 stackstate-2019-12-17-1.log.gz
-rw-r--r--. 1 stackstate stackstate  2941900 Dec 18 10:27 stackstate_deprecation.json
-rw-r--r--. 1 stackstate stackstate  1932090 Dec 18 10:27 stackstate_deprecation.log
-rw-r--r--. 1 stackstate stackstate   145330 Dec 18 12:54 stackstate.log
-rw-r--r--. 1 stackstate stackstate   344879 Dec 18 12:54 stackstate_server.json

./kafka:
-rw-r--r--. 1 stackstate stackstate        0 Oct 15 15:46 authorizer.log
-rw-r--r--. 1 stackstate stackstate   538780 Dec 18 12:50 controller.log
-rw-r--r--. 1 stackstate stackstate 11640758 Nov 20 10:04 kafka-gc.log
-rw-r--r--. 1 stackstate stackstate    15856 Dec 17 16:02 log-cleaner.log
-rw-r--r--. 1 stackstate stackstate        0 Oct 15 15:46 request.log
-rw-r--r--. 1 stackstate stackstate   397450 Dec 18 12:45 server.log
-rw-r--r--. 1 stackstate stackstate   204722 Dec 17 13:45 state-change.log

./kafka-to-es:
-rw-r--r--. 1 stackstate stackstate   180046 Dec 18 12:54 kafkaEventsToES.log
-rw-r--r--. 1 stackstate stackstate   202857 Dec 17 23:58 kafkaEventsToES.log.2019-12-17.0
-rw-r--r--. 1 stackstate stackstate    13164 Dec 18 12:23 kafkaMultiMetricsToES.log
-rw-r--r--. 1 stackstate stackstate    81363 Dec 17 23:55 kafkaMultiMetricsToES.log.2019-12-17.0
-rw-r--r--. 1 stackstate stackstate    19358 Dec 18 12:18 kafkaStsEventsToES.log
-rw-r--r--. 1 stackstate stackstate   141928 Dec 17 23:52 kafkaStsEventsToES.log.2019-12-17.0

./license-check:
-rw-r--r--. 1 stackstate stackstate 4928 Nov 14 12:39 license-app.log

./stackgraph:

./stackpack:
-rw-r--r--. 1 stackstate stackstate   906 Dec 13 13:57 agent-common.2019-12-13.0.log
-rw-r--r--. 1 stackstate stackstate  4990 Dec 16 09:11 agent-common.log
-rw-r--r--. 1 stackstate stackstate  1818 Dec 16 09:03 aws.log
-rw-r--r--. 1 stackstate stackstate  3161 Dec 13 14:02 common.2019-12-13.0.log
-rw-r--r--. 1 stackstate stackstate  5418 Dec 16 09:11 common.log
-rw-r--r--. 1 stackstate stackstate  1354 Dec 13 14:02 k8s-common.log
-rw-r--r--. 1 stackstate stackstate  4274 Dec 13 14:03 kubernetes.log
-rw-r--r--. 1 stackstate stackstate   533 Oct 21 09:26 stackstate-agent-v2.2019-10-21.0.log
-rw-r--r--. 1 stackstate stackstate 72850 Dec 16 09:11 stackstate-agent-v2.log

./stackstate:

./stackstate-receiver:
-rw-r--r--. 1 stackstate stackstate 29430771 Dec 17 23:59 stackstate-receiver.2019-12-17.0.log
-rw-r--r--. 1 stackstate stackstate 11586394 Dec 18 12:54 stackstate-receiver.log

./sync:
total 1043088
-rw-r--r--. 1 stackstate stackstate 24768357 Dec 17 23:59 sync.Agent.2019-12-17.9.log
-rw-r--r--. 1 stackstate stackstate 52430757 Dec 18 02:31 sync.Agent.2019-12-18.0.log
-rw-r--r--. 1 stackstate stackstate 52428990 Dec 18 05:11 sync.Agent.2019-12-18.1.log
-rw-r--r--. 1 stackstate stackstate 52430847 Dec 18 07:48 sync.Agent.2019-12-18.2.log
-rw-r--r--. 1 stackstate stackstate 52430381 Dec 18 10:26 sync.Agent.2019-12-18.3.log
-rw-r--r--. 1 stackstate stackstate 45954315 Dec 18 12:54 sync.Agent.log
-rw-r--r--. 1 stackstate stackstate   383053 Dec 17 23:13 sync.AWS Sync for Account - 508573134510.2019-12-17.0.log
-rw-r--r--. 1 stackstate stackstate   160967 Dec 18 12:13 sync.AWS Sync for Account - 508573134510.log
-rw-r--r--. 1 stackstate stackstate   248557 Dec 17 23:30 sync.AWS Sync for Account - 766509113410.2019-12-17.0.log
-rw-r--r--. 1 stackstate stackstate   120016 Dec 18 12:30 sync.AWS Sync for Account - 766509113410.log
-rw-r--r--. 1 stackstate stackstate 52494635 Dec 18 05:12 sync.Kubernetes - stseuw1-preprod-dev-eks-dev.2019-12-18.1.log
-rw-r--r--. 1 stackstate stackstate 52490024 Dec 18 07:48 sync.Kubernetes - stseuw1-preprod-dev-eks-dev.2019-12-18.2.log
-rw-r--r--. 1 stackstate stackstate 52450773 Dec 18 10:22 sync.Kubernetes - stseuw1-preprod-dev-eks-dev.2019-12-18.3.log
-rw-r--r--. 1 stackstate stackstate 52478535 Dec 18 12:49 sync.Kubernetes - stseuw1-preprod-dev-eks-dev.2019-12-18.4.log
-rw-r--r--. 1 stackstate stackstate  2014902 Dec 18 12:54 sync.Kubernetes - stseuw1-preprod-dev-eks-dev.log
-rw-r--r--. 1 stackstate stackstate 52438711 Dec 17 20:39 sync.Kubernetes - stseuw1-sandbox-main-eks-sandbox.2019-12-17.5.log
-rw-r--r--. 1 stackstate stackstate 50769208 Dec 17 23:59 sync.Kubernetes - stseuw1-sandbox-main-eks-sandbox.2019-12-17.6.log
-rw-r--r--. 1 stackstate stackstate 52497481 Dec 18 03:26 sync.Kubernetes - stseuw1-sandbox-main-eks-sandbox.2019-12-18.0.log
-rw-r--r--. 1 stackstate stackstate 52530973 Dec 18 06:53 sync.Kubernetes - stseuw1-sandbox-main-eks-sandbox.2019-12-18.1.log
-rw-r--r--. 1 stackstate stackstate 52539980 Dec 18 10:20 sync.Kubernetes - stseuw1-sandbox-main-eks-sandbox.2019-12-18.2.log
-rw-r--r--. 1 stackstate stackstate 39488493 Dec 18 12:54 sync.Kubernetes - stseuw1-sandbox-main-eks-sandbox.log
-rw-r--r--. 1 stackstate stackstate 52435381 Dec 17 15:23 sync.Kubernetes - stseuw1-tooling-main-eks-tooling.2019-12-17.2.log
-rw-r--r--. 1 stackstate stackstate 52429788 Dec 17 20:32 sync.Kubernetes - stseuw1-tooling-main-eks-tooling.2019-12-17.3.log
-rw-r--r--. 1 stackstate stackstate 35254609 Dec 17 23:59 sync.Kubernetes - stseuw1-tooling-main-eks-tooling.2019-12-17.4.log
-rw-r--r--. 1 stackstate stackstate 52435568 Dec 18 05:08 sync.Kubernetes - stseuw1-tooling-main-eks-tooling.2019-12-18.0.log
-rw-r--r--. 1 stackstate stackstate 52456809 Dec 18 10:14 sync.Kubernetes - stseuw1-tooling-main-eks-tooling.2019-12-18.1.log
-rw-r--r--. 1 stackstate stackstate 27479646 Dec 18 12:54 sync.Kubernetes - stseuw1-tooling-main-eks-tooling.log
-rw-r--r--. 1 stackstate stackstate      800 Dec 16 09:19 sync.Mysql.2019-12-16.0.log
-rw-r--r--. 1 stackstate stackstate      798 Dec 17 13:52 sync.Mysql.log
-rw-r--r--. 1 stackstate stackstate      805 Dec 16 09:19 sync.Postgresql.2019-12-16.0.log
-rw-r--r--. 1 stackstate stackstate      805 Dec 17 13:52 sync.Postgresql.log

From StackState v1.15.0, the version of Elasticsearch used by StackState changed. Elasticsearch logs are now saved in ./elasticsearch7. The old ./elasticsearch subdirectory can be removed to free some disk space.

Files

StackState keeps logs in files that have a maximum size of 50 MB. When a log file exceeds 50 MB size cap, it is divided into ordered parts, as in the below example:

./sync:
total 1043088
-rw-r--r--. 1 stackstate stackstate 52430757 Dec 18 02:31 sync.Agent.2019-12-18.0.log
-rw-r--r--. 1 stackstate stackstate 52428990 Dec 18 05:11 sync.Agent.2019-12-18.1.log
-rw-r--r--. 1 stackstate stackstate 52430847 Dec 18 07:48 sync.Agent.2019-12-18.2.log
-rw-r--r--. 1 stackstate stackstate 52430381 Dec 18 10:26 sync.Agent.2019-12-18.3.log

Default log pattern

StackState builds log files using the following default log pattern: "%date [%thread] %-5level %logger{60} - %msg%n"