๐Ÿ’ 
StackState Agent
StackState core integration
This page describes StackState version 4.3.

What is the Agent StackPack?

The StackState Agent provides the following functionality:
  • Reporting hosts, processes and containers
  • Reporting all network connections between processes/containers including network traffic telemetry
  • Telemetry for hosts, processes, and containers
  • 100+ additional integrations
The StackState Agent is open source and available on github at https://github.com/StackVista/stackstate-agent.

Supported configurations

StackState Agent is tested to run on the platforms listed below. Note that for Linux platforms, host data for network connections between processes and containers (including network traffic telemetry) can only be retrieved for OS versions with a network tracer (kernel version 4.3.0 or higher):
Platform
Minimum version
Notes
CentOS
CentOS 6
CentOS 6 requires Agent v2.0.2 or above. Network tracer available from CentOS 8.
Debian
Debian 7 (Wheezy)
Debian 7 (Wheezy) requires glibc upgrade to 2.17. Network tracer available from Debian 9 (Stretch).
Fedora
Fedora 28
-
RHEL
RHEL 7
Network tracer available from RHEL 8.
Ubuntu
Ubuntu 14 (Trusty)
Network tracer available from Ubuntu 16 (Xenial).
Windows
Windows 10
-
Windows Server
Windows Server 2012
-

Installation

Install the StackState Agent with one of the following commands:

Linux

Using cURLโ€‹
1
curl -o- https://stackstate-agent-2.s3.amazonaws.com/install.sh | \
2
STS_API_KEY="{{config.apiKey}}" \
3
STS_URL="{{config.baseUrl}}/stsAgent" bash
Copied!
Using wgetโ€‹
1
wget -qO- https://stackstate-agent-2.s3.amazonaws.com/install.sh | \
2
STS_API_KEY="{{config.apiKey}}" \
3
STS_URL="{{config.baseUrl}}/stsAgent" bash
Copied!

Linux offline installation

On your host, download the installation script from https://stackstate-agent-2.s3.amazonaws.com/install.sh.
By default, installer tries to configure the package update channel, which would allow to update packages using the host package manager. If you for any reason do not want this behavior, please include STS_INSTALL_NO_REPO=yes as an environment parameter:
1
STS_API_KEY="{{config.apiKey}}" \
2
STS_URL="{{config.baseUrl}}/stsAgent" \
3
STS_INSTALL_NO_REPO=yes \
4
./install.sh PATH_TO_PREDOWNLOADED_INSTALLER_PACKAGE
Copied!

Windows

Using PowerShellโ€‹
1
. { iwr -useb https://stackstate-agent-2.s3.amazonaws.com/install.ps1 } | iex; `
2
install -stsApiKey "{{config.apiKey}}" `
3
-stsUrl "{{config.baseUrl}}/stsAgent"
Copied!

Windows offline installation

On your host, download a copy of the PowerShell script from https://stackstate-agent-2.s3.amazonaws.com/install.ps1 alongside with the agent installer in the form .msi. The latest version of the installer can be downloaded from https://stackstate-agent-2.s3.amazonaws.com/windows/stable/stackstate-agent-latest-1-x86_64.msi.
Assuming your installer is saved as C:\stackstate-custom.msi, and the PowerShell script saved as C:\install_script.ps1, open PowerShell with elevated privileges and invoke the following set of commands:
1
Import-Module C:\install_script.ps1
2
install -stsApiKey {{config.apiKey}} `
3
-stsUrl {{config.baseUrl}}/stsAgent `
4
-f C:\\stackstate-custom.msi
Copied!

Docker

Installation

To run the StackState Agent as a docker container, use the following configuration:
1
stackstate-agent:
2
image: docker.io/stackstate/stackstate-agent-2:latest
3
network_mode: "host"
4
pid: "host"
5
privileged: true
6
volumes:
7
- "/var/run/docker.sock:/var/run/docker.sock:ro"
8
- "/proc/:/host/proc/:ro"
9
- "/sys/fs/cgroup/:/host/sys/fs/cgroup:ro"
10
- "/etc/passwd:/etc/passwd:ro"
11
- "/sys/kernel/debug:/sys/kernel/debug"
12
environment:
13
STS_API_KEY: "API_KEY"
14
STS_STS_URL: "https://your.stackstate.url/receiver/stsAgent"
15
STS_PROCESS_AGENT_URL: "https://your.stackstate.url/receiver/stsAgent"
16
STS_PROCESS_AGENT_ENABLED: "true"
17
STS_NETWORK_TRACING_ENABLED: "true"
18
STS_APM_URL: "https://your.stackstate.url/receiver/stsAgent"
19
STS_APM_ENABLED: "true"
20
HOST_PROC: "/host/proc"
21
HOST_SYS: "/host/sys"
Copied!

Using Docker-Swarm mode

To run the StackState Agent in Docker-Swarm mode as a docker-compose setup, use the above configuration in your compose file on each node where you want to run the Agent. After placing the compose file on each node, run the command docker-compose up -d.
Limitation of Docker-Swarm mode
Some specific features are not supported in Docker-Swarm mode. This limitation prevents StackState Agent from collecting relations between Containers, Processes and other resources while in Docker-Swarm mode. To run StackState Agent in Docker-Swarm mode, use a docker-compose setup.

Using Self-Signed Certificate

When checks are being configured to use a self-signed certificate for https requests, then the following environment variable should be overwritten:
1
CURL_CA_BUNDLE = ""
Copied!

Advanced Configurations

Process reported by the StackState Agent can be filtered using a blacklist. Using it in conjunction with the inclusion rules will include otherwise excluded processes.
Parameter
Mandatory
Default Value
Description
STS_PROCESS_BLACKLIST_PATTERNS
No
โ€‹See Githubโ€‹
A list of regex patterns that will exclude a process if matched
STS_PROCESS_BLACKLIST_INCLUSIONS_TOP_CPU
No
0
Number of processes to report that have a high CPU usage
STS_PROCESS_BLACKLIST_INCLUSIONS_TOP_IO_READ
No
0
Number of processes to report that have a high IO read usage
STS_PROCESS_BLACKLIST_INCLUSIONS_TOP_IO_WRITE
No
0
Number of processes to report that have a high IO write usage
STS_PROCESS_BLACKLIST_INCLUSIONS_TOP_MEM
No
0
Number of processes to report that have a high Memory usage
STS_PROCESS_BLACKLIST_INCLUSIONS_CPU_THRESHOLD
No
โ€‹
Threshold that enables the reporting of high CPU usage processes
STS_PROCESS_BLACKLIST_INCLUSIONS_MEM_THRESHOLD
No
โ€‹
Threshold that enables the reporting of high Memory usage processes
Certain features of the agent can be turned off if not needed:
Parameter
Mandatory
Default Value
Description
STS_PROCESS_AGENT_ENABLED
No
True
Whenever process agent should be enabled
STS_APM_ENABLED
No
True
Whenever trace agent should be enabled
STS_NETWORK_TRACING_ENABLED
No
True
Whenever network tracer should be enabled

Troubleshooting

To troubleshoot the StackState Agent container, set the logging level to debug using the STS_LOG_LEVEL environment variable:
1
STS_LOG_LEVEL: "DEBUG"
Copied!

Configuration files

Linux
Windows
  • StackState Agent configuration: /etc/stackstate-agent/stackstate.yaml
  • Integration configurations: /etc/stackstate-agent/conf.d/
  • StackState Agent configuration: C:\ProgramData\StackState\stackstate.yaml
  • Integration configurations:C:\ProgramData\StackState\conf.d

Troubleshooting

Try running the status command to see the state of the StackState Agent.

Log files

Linux
Windows
Logs for the subsystems are in the following files:
1
/var/log/stackstate-agent/agent.log
2
/var/log/stackstate-agent/process-agent.log
Copied!
Logs for the subsystems are in the following files:
1
C:\ProgramData\StackState\logs\agent.log
2
C:\ProgramData\StackState\logs\process-agent.log
Copied!

Start / stop / restart the StackState Agent

  • Commands require elevated privileges.
  • Restarting the StackState Agent will reload the configuration files.
To manually start, stop or restart the StackState Agent:
Linux
Windows
1
sudo service stackstate-agent start
2
sudo service stackstate-agent stop
3
sudo service stackstate-agent restart
Copied!
CMD
1
"C:\Program Files\StackState\StackState Agent\embedded\agent.exe" start-service
2
"C:\Program Files\StackState\StackState Agent\embedded\agent.exe" stopservice
3
"C:\Program Files\StackState\StackState Agent\embedded\agent.exe" restart-service
Copied!
PowerShell
1
& "C:\Program Files\StackState\StackState Agent\embedded\agent.exe" start-service
2
& "C:\Program Files\StackState\StackState Agent\embedded\agent.exe" stopservice
3
& "C:\Program Files\StackState\StackState Agent\embedded\agent.exe" restart-service
Copied!

Status and information

Linux
Windows
To check if the StackState Agent is running and receive information about the Agent's state:
1
sudo service stackstate-agent status
Copied!
Tracebacks for errors can be retrieved by setting the -v flag:
1
sudo service stackstate-agent status -v
Copied!
To check if the StackState Agent is running and receive information about the Agent's state:
1
"./agent.exe status"
Copied!

Agent overhead

StackState Agent V2 consists of up to four different processes - stackstate-agent, trace-agent, process-agent and cluster-agent. To run the basic Agent, the resources named below are required. These were observed running StackState Agent V2 v2.13.0 on a c5.xlarge instance with 4 vCPU cores and 8GB RAM. They give an indication of the overhead for the most simple set up. Actual resource usage will increase based on the Agent configuration running. This can be impacted by factors such as the Agent processes that are enabled, the number and nature of checks running, whether network connection tracking and protocol inspection are enabled, and the number of Kubernetes pods from which metrics are collected on the same host as the Agent.
stackstate-agent
process-agent
trace-agent
Resource
Usage
CPU
~0.18%
Memory
95-100MB RAM
Disk space
461MB (includes stackstate-agent, process-agent and trace-agent)
Resource
Usage
CPU
up to 0.96%
Memory
52-56MB
Disk space
461MB (includes stackstate-agent, process-agent and trace-agent)
Resource
Usage
CPU
less than 0.04%
Memory
less than 16.8MB
Disk space
461MB (includes stackstate-agent, process-agent and trace-agent)
On Kubernetes, limits are placed on CPU and memory usage of the Agent, Cluster Agent and Cluster checks. These can be configured in the Agent Helm chart (github.com).

Release notes

Agent V2 StackPack v4.3.1 (2021-04-02)
  • Features: Introduced swarm services as components and relations with containers.
  • Features: Report desired replicas and active replicas for swarm services.
  • Features: Health check added for swarm service on active replicas.
  • Improvement: Enable auto grouping on generated views.
  • Improvement: Common bumped from 2.3.1 to 2.5.0
  • Improvement: StackState min version bumped to 4.3.0
Agent V2 StackPack v4.2.1 (2021-03-11)
  • Bugfix: Fix for trace service types causing spurious updates on StackState.
Agent V2 StackPack v4.2.0 (2021-02-26)
  • Features: Map the container restart event stream as metric stream.
  • Features: Introduced the container health check for restart event.
  • Features: Introduced Disk Metrics and Check on Host in Agent V2 StackPack.
  • Features: Separate Sync and DataSource added for Disk Type.
Agent V2 StackPack v4.1.0 (2021-02-08)
  • Improvement: Updated the "Agent Container Mapping Function" and "Agent Container Template" to map the container name instead of the container id to the identifier
  • Bugfix: Fix the error stream for the traces not coming from traefik.
Agent V2 StackPack v4.0.0 (2021-01-29)
  • Bugfix: Major bump the version for installation fix
Agent V2 StackPack v3.12.0 (2020-12-15)
  • Feature: Split error types in traces into:
    • 5xx errors - Use this in check function to determine critical status in the component.
    • 4xx errors.
Agent V2 StackPack v3.11.0 (2020-09-03)
  • Feature: Added the Agent Integration synchronization, mapping functions and templates to synchronize topology and telemetry coming from custom Agent Integrations.
  • Feature: Added the "Create your own" integration StackPack page that explains how to build a custom integration in the StackState Agent.
  • Feature: Introduced monitoring of all StackState Agent Integrations in the Agent - Integrations - All View.
Agent V2 StackPack v3.10.1 (2020-08-18)
  • Feature: Introduced the Release notes pop up for customer.
  • Feature: Introduced the Docker-Swarm mode setup docs in Docker integration.
Last modified 2mo ago