StackState log files
Work with the StackState log files on Kubernetes and Linux

Kubernetes

In a Kubernetes setup, StackState functions are distributed across different pods and logs for each function are stored per pod and container. You can access recent logs using kubectl, although for long term storage it is recommended to set up log aggregation.

Access recent logs

The most recent logs can be retrieved from Kubernetes using the kubectl logs command. For example:
1
# Snapshot of logs for all containers of <pod-name>
2
kubectl logs <pod-name> --all-containers=true
3
​
4
# Stream logs for all containers of <pod-name>
5
kubectl logs -f <pod-name> --all-containers=true
6
​
7
# Snapshot of logs for a specific container of <pod-name>
8
kubectl logs -c <container-name> <pod-name>
9
​
10
# Snapshot of logs for previous terminated container of <pod-name>
11
kubectl logs -p -c <container-name> <pod-name>
Copied!
You can use the synchronization name to locate specific log information in a log snapshot. For example:
1
# Logs of the synchronization for a specific Kubernetes cluster
2
kubectl logs stackstate-sync-0 | grep "Kubernetes - \<cluster-name\>"
3
​
4
# Logs of the agent synchronization
5
kubectl logs stackstate-sync-0 | grep "Agent"
Copied!

Find StackState logs on Kubernetes pods

StackState logs are stored per pod and container. The table below shows the pod to access for logs relating to specific StackState functions. Note that actual pod names will include a number or random string suffix (e.g. stackstate-receiver-5b9d79db86-h2hkz) and may also include the release name specified when StackState was deployed as a prefix.
StackState function
Pod
API (including topology, charts and settings)
stackstate-api
Checks
stackstate-checks
Data indexing into Elasticsearch
stackstate-mm2es (metrics) stackstate-e2es (events) stackstate-trace2es (traces) stackstate-sts2es (events generated by StackState)
Data ingestion
stackstate-receiver
Event handlers
stackstate-view-health
State propagation
stackstate-state
Synchronization
stackstate-sync
View health state
stackstate-view-health
You can access logs on a specific pod using the kubectl logs command. For example:
1
kubectl logs stackstate-api-0
Copied!
Note that logs stored on pods will be regularly removed. For long term access to logs, it is advised that you set up log aggregation for your Kubernetes cluster.

Log aggregation

For long term storage of StackState log data, it is advised that you set up log aggregation on your Kubernetes cluster. This can be done using a third party system for storage such as Elasticsearch, Splunk or Logz.io and a log shipper such as Logstash or Fluentd.
For more details of how this can be done, check:

Linux

In a Linux setup, StackState keeps all log files in the var/log subdirectory of the StackState installation directory. By default this is /opt/stackstate/var/log. In case of a two-node installation, logs are kept in the var/log directory on each node. Note that the logs are node-specific - the StackState node keeps StackState related logs and the StackGraph node keeps logs related to StackGraph.

var/log details

Inside var/log, there is a subdirectory for each type of logs kept by StackState. By default, each subdirectory has a size cap of 2 GB; when exceeded, the oldest files are deleted. Example of var/log contents:
1
.:
2
drwxr-xr-x. 2 stackstate stackstate 4096 Dec 18 00:00 correlate
3
drwxr-xr-x. 2 stackstate stackstate 4096 Oct 23 00:00 elasticsearch
4
drwxr-xr-x. 2 stackstate stackstate 12288 Dec 18 00:00 elasticsearch7
5
drwxr-xr-x. 2 stackstate stackstate 4096 Dec 18 00:00 kafka
6
drwxr-xr-x. 2 stackstate stackstate 4096 Dec 18 00:20 kafka-to-es
7
drwxr-xr-x. 2 stackstate stackstate 4096 Nov 14 12:35 license-check
8
-rw-r--r--. 1 stackstate stackstate 2987 Dec 17 13:43 processmanager.2019-12-16.0.log
9
-rw-r--r--. 1 stackstate stackstate 65240 Dec 17 23:55 processmanager.2019-12-17.0.log
10
-rw-r--r--. 1 stackstate stackstate 10656 Dec 18 12:03 processmanager.log
11
drwxr-xr-x. 2 stackstate stackstate 4096 Oct 15 15:25 stackgraph
12
drwxr-xr-x. 2 stackstate stackstate 4096 Dec 16 09:03 stackpack
13
drwxr-xr-x. 2 stackstate stackstate 4096 Dec 17 13:21 stackstate
14
-rw-r--r--. 1 stackstate stackstate 6136841 Dec 16 23:59 stackstate.2019-12-16.11.log
15
-rw-r--r--. 1 stackstate stackstate 52429560 Dec 17 12:02 stackstate.2019-12-17.0.log
16
-rw-r--r--. 1 stackstate stackstate 52429586 Dec 17 20:39 stackstate.2019-12-17.1.log
17
-rw-r--r--. 1 stackstate stackstate 20252875 Dec 17 23:59 stackstate.2019-12-17.2.log
18
-rw-r--r--. 1 stackstate stackstate 52449712 Dec 18 00:37 stackstate.2019-12-18.0.log
19
-rw-r--r--. 1 stackstate stackstate 52441741 Dec 18 06:22 stackstate.2019-12-18.10.log
20
-rw-r--r--. 1 stackstate stackstate 52450906 Dec 18 07:27 stackstate.2019-12-18.11.log
21
-rw-r--r--. 1 stackstate stackstate 52478100 Dec 18 08:33 stackstate.2019-12-18.12.log
22
-rw-r--r--. 1 stackstate stackstate 52490055 Dec 18 09:37 stackstate.2019-12-18.13.log
23
-rw-r--r--. 1 stackstate stackstate 52428811 Dec 18 10:42 stackstate.2019-12-18.14.log
24
-rw-r--r--. 1 stackstate stackstate 52496521 Dec 18 11:48 stackstate.2019-12-18.15.log
25
-rw-r--r--. 1 stackstate stackstate 52445570 Dec 18 12:54 stackstate.2019-12-18.16.log
26
-rw-r--r--. 1 stackstate stackstate 52435555 Dec 18 01:10 stackstate.2019-12-18.1.log
27
-rw-r--r--. 1 stackstate stackstate 52453947 Dec 18 01:44 stackstate.2019-12-18.2.log
28
-rw-r--r--. 1 stackstate stackstate 52449896 Dec 18 02:19 stackstate.2019-12-18.3.log
29
-rw-r--r--. 1 stackstate stackstate 52429347 Dec 18 02:54 stackstate.2019-12-18.4.log
30
-rw-r--r--. 1 stackstate stackstate 52435159 Dec 18 03:29 stackstate.2019-12-18.5.log
31
-rw-r--r--. 1 stackstate stackstate 52486191 Dec 18 04:03 stackstate.2019-12-18.6.log
32
-rw-r--r--. 1 stackstate stackstate 52440210 Dec 18 04:38 stackstate.2019-12-18.7.log
33
-rw-r--r--. 1 stackstate stackstate 52436912 Dec 18 05:13 stackstate.2019-12-18.8.log
34
-rw-r--r--. 1 stackstate stackstate 52432884 Dec 18 05:47 stackstate.2019-12-18.9.log
35
-rw-r--r--. 1 stackstate stackstate 469460 Dec 18 12:54 stackstate.log
36
drwxr-xr-x. 2 stackstate stackstate 4096 Dec 18 00:00 stackstate-receiver
37
-rw-r--r--. 1 stackstate stackstate 1067419 Nov 20 10:04 stackstate-service.log
38
drwxr-xr-x. 2 stackstate stackstate 4096 Dec 18 12:49 sync
39
-rw-r--r--. 1 stackstate stackstate 52437278 Dec 18 12:34 transaction.metrics.2019-12-18.100.log
40
-rw-r--r--. 1 stackstate stackstate 52474749 Dec 18 12:41 transaction.metrics.2019-12-18.101.log
41
-rw-r--r--. 1 stackstate stackstate 52471248 Dec 18 12:49 transaction.metrics.2019-12-18.102.log
42
-rw-r--r--. 1 stackstate stackstate 52432268 Dec 18 10:22 transaction.metrics.2019-12-18.83.log
43
-rw-r--r--. 1 stackstate stackstate 52455578 Dec 18 10:30 transaction.metrics.2019-12-18.84.log
44
-rw-r--r--. 1 stackstate stackstate 52435647 Dec 18 10:39 transaction.metrics.2019-12-18.85.log
45
-rw-r--r--. 1 stackstate stackstate 52548076 Dec 18 10:49 transaction.metrics.2019-12-18.86.log
46
-rw-r--r--. 1 stackstate stackstate 52434274 Dec 18 10:57 transaction.metrics.2019-12-18.87.log
47
-rw-r--r--. 1 stackstate stackstate 52537926 Dec 18 11:05 transaction.metrics.2019-12-18.88.log
48
-rw-r--r--. 1 stackstate stackstate 52460399 Dec 18 11:12 transaction.metrics.2019-12-18.89.log
49
-rw-r--r--. 1 stackstate stackstate 52455265 Dec 18 11:19 transaction.metrics.2019-12-18.90.log
50
-rw-r--r--. 1 stackstate stackstate 52452737 Dec 18 11:26 transaction.metrics.2019-12-18.91.log
51
-rw-r--r--. 1 stackstate stackstate 52450217 Dec 18 11:34 transaction.metrics.2019-12-18.92.log
52
-rw-r--r--. 1 stackstate stackstate 52430615 Dec 18 11:41 transaction.metrics.2019-12-18.93.log
53
-rw-r--r--. 1 stackstate stackstate 52466370 Dec 18 11:48 transaction.metrics.2019-12-18.94.log
54
-rw-r--r--. 1 stackstate stackstate 52449864 Dec 18 11:55 transaction.metrics.2019-12-18.95.log
55
-rw-r--r--. 1 stackstate stackstate 52430948 Dec 18 12:03 transaction.metrics.2019-12-18.96.log
56
-rw-r--r--. 1 stackstate stackstate 52442265 Dec 18 12:11 transaction.metrics.2019-12-18.97.log
57
-rw-r--r--. 1 stackstate stackstate 52502949 Dec 18 12:19 transaction.metrics.2019-12-18.98.log
58
-rw-r--r--. 1 stackstate stackstate 52479825 Dec 18 12:26 transaction.metrics.2019-12-18.99.log
59
-rw-r--r--. 1 stackstate stackstate 44666747 Dec 18 12:54 transaction.metrics.log
60
​
61
./correlate:
62
-rw-r--r--. 1 stackstate stackstate 10389822 Dec 18 12:54 correlate.log
63
-rw-r--r--. 1 stackstate stackstate 3913232 Dec 18 12:54 worker.correlateConnections-1.log
64
-rw-r--r--. 1 stackstate stackstate 2629015 Dec 18 12:54 worker.correlateConnections-2.log
65
-rw-r--r--. 1 stackstate stackstate 3731346 Dec 18 12:54 worker.correlateConnections-3.log
66
​
67
./elasticsearch:
68
-rw-r--r--. 1 stackstate stackstate 3401 Oct 15 15:46 stackstate-2019-10-15.log
69
-rw-r--r--. 1 stackstate stackstate 3484 Oct 16 08:27 stackstate-2019-10-16.log
70
-rw-r--r--. 1 stackstate stackstate 50560 Oct 17 08:24 stackstate-2019-10-17.log
71
-rw-r--r--. 1 stackstate stackstate 3678 Oct 18 07:01 stackstate-2019-10-18.log
72
-rw-r--r--. 1 stackstate stackstate 32024 Oct 21 21:05 stackstate-2019-10-21.log
73
-rw-r--r--. 1 stackstate stackstate 129371 Oct 22 23:19 stackstate-2019-10-22.log
74
-rw-r--r--. 1 stackstate stackstate 0 Oct 15 15:46 stackstate_deprecation.log
75
-rw-r--r--. 1 stackstate stackstate 0 Oct 15 15:46 stackstate_index_indexing_slowlog.log
76
-rw-r--r--. 1 stackstate stackstate 0 Oct 15 15:46 stackstate_index_search_slowlog.log
77
-rw-r--r--. 1 stackstate stackstate 4664 Oct 23 10:29 stackstate.log
78
​
79
./elasticsearch7:
80
-rw-r--r--. 1 stackstate stackstate 75279 Dec 18 00:00 stackstate-2019-12-17-1.json.gz
81
-rw-r--r--. 1 stackstate stackstate 70897 Dec 18 00:00 stackstate-2019-12-17-1.log.gz
82
-rw-r--r--. 1 stackstate stackstate 2941900 Dec 18 10:27 stackstate_deprecation.json
83
-rw-r--r--. 1 stackstate stackstate 1932090 Dec 18 10:27 stackstate_deprecation.log
84
-rw-r--r--. 1 stackstate stackstate 145330 Dec 18 12:54 stackstate.log
85
-rw-r--r--. 1 stackstate stackstate 344879 Dec 18 12:54 stackstate_server.json
86
​
87
./kafka:
88
-rw-r--r--. 1 stackstate stackstate 0 Oct 15 15:46 authorizer.log
89
-rw-r--r--. 1 stackstate stackstate 538780 Dec 18 12:50 controller.log
90
-rw-r--r--. 1 stackstate stackstate 11640758 Nov 20 10:04 kafka-gc.log
91
-rw-r--r--. 1 stackstate stackstate 15856 Dec 17 16:02 log-cleaner.log
92
-rw-r--r--. 1 stackstate stackstate 0 Oct 15 15:46 request.log
93
-rw-r--r--. 1 stackstate stackstate 397450 Dec 18 12:45 server.log
94
-rw-r--r--. 1 stackstate stackstate 204722 Dec 17 13:45 state-change.log
95
​
96
./kafka-to-es:
97
-rw-r--r--. 1 stackstate stackstate 180046 Dec 18 12:54 kafkaEventsToES.log
98
-rw-r--r--. 1 stackstate stackstate 202857 Dec 17 23:58 kafkaEventsToES.log.2019-12-17.0
99
-rw-r--r--. 1 stackstate stackstate 13164 Dec 18 12:23 kafkaMultiMetricsToES.log
100
-rw-r--r--. 1 stackstate stackstate 81363 Dec 17 23:55 kafkaMultiMetricsToES.log.2019-12-17.0
101
-rw-r--r--. 1 stackstate stackstate 19358 Dec 18 12:18 kafkaStsEventsToES.log
102
-rw-r--r--. 1 stackstate stackstate 141928 Dec 17 23:52 kafkaStsEventsToES.log.2019-12-17.0
103
​
104
./license-check:
105
-rw-r--r--. 1 stackstate stackstate 4928 Nov 14 12:39 license-app.log
106
​
107
./stackgraph:
108
​
109
./stackpack:
110
-rw-r--r--. 1 stackstate stackstate 906 Dec 13 13:57 agent-common.2019-12-13.0.log
111
-rw-r--r--. 1 stackstate stackstate 4990 Dec 16 09:11 agent-common.log
112
-rw-r--r--. 1 stackstate stackstate 1818 Dec 16 09:03 aws.log
113
-rw-r--r--. 1 stackstate stackstate 3161 Dec 13 14:02 common.2019-12-13.0.log
114
-rw-r--r--. 1 stackstate stackstate 5418 Dec 16 09:11 common.log
115
-rw-r--r--. 1 stackstate stackstate 1354 Dec 13 14:02 k8s-common.log
116
-rw-r--r--. 1 stackstate stackstate 4274 Dec 13 14:03 kubernetes.log
117
-rw-r--r--. 1 stackstate stackstate 533 Oct 21 09:26 stackstate-agent-v2.2019-10-21.0.log
118
-rw-r--r--. 1 stackstate stackstate 72850 Dec 16 09:11 stackstate-agent-v2.log
119
​
120
./stackstate:
121
​
122
./stackstate-receiver:
123
-rw-r--r--. 1 stackstate stackstate 29430771 Dec 17 23:59 stackstate-receiver.2019-12-17.0.log
124
-rw-r--r--. 1 stackstate stackstate 11586394 Dec 18 12:54 stackstate-receiver.log
125
​
126
./sync:
127
total 1043088
128
-rw-r--r--. 1 stackstate stackstate 24768357 Dec 17 23:59 sync.Agent.2019-12-17.9.log
129
-rw-r--r--. 1 stackstate stackstate 52430757 Dec 18 02:31 sync.Agent.2019-12-18.0.log
130
-rw-r--r--. 1 stackstate stackstate 52428990 Dec 18 05:11 sync.Agent.2019-12-18.1.log
131
-rw-r--r--. 1 stackstate stackstate 52430847 Dec 18 07:48 sync.Agent.2019-12-18.2.log
132
-rw-r--r--. 1 stackstate stackstate 52430381 Dec 18 10:26 sync.Agent.2019-12-18.3.log
133
-rw-r--r--. 1 stackstate stackstate 45954315 Dec 18 12:54 sync.Agent.log
134
-rw-r--r--. 1 stackstate stackstate 383053 Dec 17 23:13 sync.AWS Sync for Account - 508573134510.2019-12-17.0.log
135
-rw-r--r--. 1 stackstate stackstate 160967 Dec 18 12:13 sync.AWS Sync for Account - 508573134510.log
136
-rw-r--r--. 1 stackstate stackstate 248557 Dec 17 23:30 sync.AWS Sync for Account - 766509113410.2019-12-17.0.log
137
-rw-r--r--. 1 stackstate stackstate 120016 Dec 18 12:30 sync.AWS Sync for Account - 766509113410.log
138
-rw-r--r--. 1 stackstate stackstate 52494635 Dec 18 05:12 sync.Kubernetes - stseuw1-preprod-dev-eks-dev.2019-12-18.1.log
139
-rw-r--r--. 1 stackstate stackstate 52490024 Dec 18 07:48 sync.Kubernetes - stseuw1-preprod-dev-eks-dev.2019-12-18.2.log
140
-rw-r--r--. 1 stackstate stackstate 52450773 Dec 18 10:22 sync.Kubernetes - stseuw1-preprod-dev-eks-dev.2019-12-18.3.log
141
-rw-r--r--. 1 stackstate stackstate 52478535 Dec 18 12:49 sync.Kubernetes - stseuw1-preprod-dev-eks-dev.2019-12-18.4.log
142
-rw-r--r--. 1 stackstate stackstate 2014902 Dec 18 12:54 sync.Kubernetes - stseuw1-preprod-dev-eks-dev.log
143
-rw-r--r--. 1 stackstate stackstate 52438711 Dec 17 20:39 sync.Kubernetes - stseuw1-sandbox-main-eks-sandbox.2019-12-17.5.log
144
-rw-r--r--. 1 stackstate stackstate 50769208 Dec 17 23:59 sync.Kubernetes - stseuw1-sandbox-main-eks-sandbox.2019-12-17.6.log
145
-rw-r--r--. 1 stackstate stackstate 52497481 Dec 18 03:26 sync.Kubernetes - stseuw1-sandbox-main-eks-sandbox.2019-12-18.0.log
146
-rw-r--r--. 1 stackstate stackstate 52530973 Dec 18 06:53 sync.Kubernetes - stseuw1-sandbox-main-eks-sandbox.2019-12-18.1.log
147
-rw-r--r--. 1 stackstate stackstate 52539980 Dec 18 10:20 sync.Kubernetes - stseuw1-sandbox-main-eks-sandbox.2019-12-18.2.log
148
-rw-r--r--. 1 stackstate stackstate 39488493 Dec 18 12:54 sync.Kubernetes - stseuw1-sandbox-main-eks-sandbox.log
149
-rw-r--r--. 1 stackstate stackstate 52435381 Dec 17 15:23 sync.Kubernetes - stseuw1-tooling-main-eks-tooling.2019-12-17.2.log
150
-rw-r--r--. 1 stackstate stackstate 52429788 Dec 17 20:32 sync.Kubernetes - stseuw1-tooling-main-eks-tooling.2019-12-17.3.log
151
-rw-r--r--. 1 stackstate stackstate 35254609 Dec 17 23:59 sync.Kubernetes - stseuw1-tooling-main-eks-tooling.2019-12-17.4.log
152
-rw-r--r--. 1 stackstate stackstate 52435568 Dec 18 05:08 sync.Kubernetes - stseuw1-tooling-main-eks-tooling.2019-12-18.0.log
153
-rw-r--r--. 1 stackstate stackstate 52456809 Dec 18 10:14 sync.Kubernetes - stseuw1-tooling-main-eks-tooling.2019-12-18.1.log
154
-rw-r--r--. 1 stackstate stackstate 27479646 Dec 18 12:54 sync.Kubernetes - stseuw1-tooling-main-eks-tooling.log
155
-rw-r--r--. 1 stackstate stackstate 800 Dec 16 09:19 sync.Mysql.2019-12-16.0.log
156
-rw-r--r--. 1 stackstate stackstate 798 Dec 17 13:52 sync.Mysql.log
157
-rw-r--r--. 1 stackstate stackstate 805 Dec 16 09:19 sync.Postgresql.2019-12-16.0.log
158
-rw-r--r--. 1 stackstate stackstate 805 Dec 17 13:52 sync.Postgresql.log
Copied!
From StackState v1.15.0, the version of Elasticsearch used by StackState changed. Elasticsearch logs are now saved in ./elasticsearch7. The old ./elasticsearch subdirectory can be removed to free some disk space.

Files

StackState keeps logs in files that have a maximum size of 50 MB. When a log file exceeds 50 MB size cap, it is divided into ordered parts, as in the below example:
1
./sync:
2
total 1043088
3
-rw-r--r--. 1 stackstate stackstate 52430757 Dec 18 02:31 sync.Agent.2019-12-18.0.log
4
-rw-r--r--. 1 stackstate stackstate 52428990 Dec 18 05:11 sync.Agent.2019-12-18.1.log
5
-rw-r--r--. 1 stackstate stackstate 52430847 Dec 18 07:48 sync.Agent.2019-12-18.2.log
6
-rw-r--r--. 1 stackstate stackstate 52430381 Dec 18 10:26 sync.Agent.2019-12-18.3.log
Copied!

Default log pattern

StackState builds log files using the following default log pattern: "%date [%thread] %-5level %logger{60} - %msg%n"
Last modified 2mo ago