Stackstate-Gunicorn Integration

Overview

The StackState Agent collects one main metric about Gunicorn: the number of worker processes running. Gunicorn itself can provide further metrics via StsStatsD, including those for:

  • Total request rate
  • Request rate by status code (2xx, 3xx, 4xx, 5xx)
  • Request duration (average, median, max, 95th percentile, etc)
  • Log message rate by log level (critical, error, warning, exception)

Setup

Installation

This check is packaged with the Agent, so simply install the agent.

The Gunicorn check requires your Gunicorn app’s Python environment to have the setproctitle package; without it, the StackState Agent will always report that it cannot find a gunicorn master process (and hence, cannot find workers, either). Install the setproctitle package in your app’s Python environment if you want to collect the gunicorn.workers metric.

Configuration

Configure the Agent

Create a gunicorn.yaml in the StackState Agent’s conf.d directory:

init_config:

instances:
  # as set
  # 1) in your app's config.py (proc_name = <YOUR_APP_NAME>), OR
  # 2) via CLI (gunicorn --name <YOUR_APP_NAME> your:app)
  - proc_name: <YOUR_APP_NAME>

Restart the Agent to begin sending Gunicorn metrics to StackState.

Connect Gunicorn to StsStatsD

Since version 19.1, Gunicorn provides an option to send its metrics to a StatsD daemon. As with many Gunicorn options, you can either pass it to gunicorn on the CLI (--statsd-host) or set it in your app’s configuration file (statsd_host). Configure your app to send metrics to StsStatsD at "localhost:8125", and restart the app.

Validation

Run the Agent’s info command and look for gunicorn under the Checks section:

Checks
======
  [...]

  gunicorn (5.12.1)
  -----------------
    - instance #0 [OK]
    - Collected 2 metrics, 0 events & 1 service check

  [...]

If the status is not OK, see the Troubleshooting section.

Use netstat to verify that Gunicorn is sending its metrics, too:

$ sudo netstat -nup | grep "127.0.0.1:8125.*ESTABLISHED"
udp        0      0 127.0.0.1:38374         127.0.0.1:8125          ESTABLISHED 15500/gunicorn: mas

Troubleshooting

Agent cannot find Gunicorn process

Checks
======

  gunicorn (5.12.1)
  -----------------
    - instance #0 [ERROR]: 'Found no master process with name: gunicorn: master [my_web_app]'
    - Collected 0 metrics, 0 events & 1 service check
    - Dependencies:
        - psutil: 4.4.1

Either Gunicorn really isn’t running, or your app’s Python environment doesn’t have the setproctitle package installed.

If setproctitle is not installed, Gunicorn appears in the process table like so:

$ ps -ef | grep gunicorn
ubuntu   18013 16695  2 20:23 pts/0    00:00:00 /usr/bin/python /usr/bin/gunicorn --config test-app-config.py gunicorn-test:app
ubuntu   18018 18013  0 20:23 pts/0    00:00:00 /usr/bin/python /usr/bin/gunicorn --config test-app-config.py gunicorn-test:app
ubuntu   18019 18013  0 20:23 pts/0    00:00:00 /usr/bin/python /usr/bin/gunicorn --config test-app-config.py gunicorn-test:app

If it is installed, gunicorn processes appear in the format the StackState Agent expects:

$ ps -ef | grep gunicorn
ubuntu   18457 16695  5 20:26 pts/0    00:00:00 gunicorn: master [my_app]
ubuntu   18462 18457  0 20:26 pts/0    00:00:00 gunicorn: worker [my_app]
ubuntu   18463 18457  0 20:26 pts/0    00:00:00 gunicorn: worker [my_app]

Data Collected

Metrics

gunicorn.requests
(rate)
The rate of requests received.
shown as request
gunicorn.workers
(gauge)
The number of workers tagged by state (idle or working).
shown as worker
gunicorn.request.duration.95percentile
(gauge)
The 95th percentile of request duration time.
shown as millisecond
gunicorn.request.duration.avg
(gauge)
The average request duration time.
shown as millisecond
gunicorn.request.duration.count
(rate)
The rate of requests received.
shown as request
gunicorn.request.duration.max
(gauge)
The maximum request duration time.
shown as millisecond
gunicorn.request.duration.median
(gauge)
The median request duration time.
shown as millisecond
gunicorn.log.critical
(rate)
The rate of logged critical statements.
shown as occurrence
gunicorn.log.error
(rate)
The rate of logged errors.
shown as occurrence
gunicorn.log.warning
(rate)
The rate of logged warnings.
shown as occurrence
gunicorn.log.exception
(rate)
The rate of logged exceptions.
shown as occurrence
gunicorn.request.status.100
(rate)
The rate of requests that generate responses with a 100 status code.
shown as request
gunicorn.request.status.101
(rate)
The rate of requests that generate responses with a 101 status code.
shown as request
gunicorn.request.status.102
(rate)
The rate of requests that generate responses with a 102 status code.
shown as request
gunicorn.request.status.200
(rate)
The rate of requests that generate responses with a 200 status code.
shown as request
gunicorn.request.status.201
(rate)
The rate of requests that generate responses with a 201 status code.
shown as request
gunicorn.request.status.202
(rate)
The rate of requests that generate responses with a 202 status code.
shown as request
gunicorn.request.status.203
(rate)
The rate of requests that generate responses with a 203 status code.
shown as request
gunicorn.request.status.204
(rate)
The rate of requests that generate responses with a 204 status code.
shown as request
gunicorn.request.status.205
(rate)
The rate of requests that generate responses with a 205 status code.
shown as request
gunicorn.request.status.206
(rate)
The rate of requests that generate responses with a 206 status code.
shown as request
gunicorn.request.status.207
(rate)
The rate of requests that generate responses with a 207 status code.
shown as request
gunicorn.request.status.208
(rate)
The rate of requests that generate responses with a 208 status code.
shown as request
gunicorn.request.status.226
(rate)
The rate of requests that generate responses with a 226 status code.
shown as request
gunicorn.request.status.300
(rate)
The rate of requests that generate responses with a 300 status code.
shown as request
gunicorn.request.status.301
(rate)
The rate of requests that generate responses with a 301 status code.
shown as request
gunicorn.request.status.302
(rate)
The rate of requests that generate responses with a 302 status code.
shown as request
gunicorn.request.status.303
(rate)
The rate of requests that generate responses with a 303 status code.
shown as request
gunicorn.request.status.304
(rate)
The rate of requests that generate responses with a 304 status code.
shown as request
gunicorn.request.status.305
(rate)
The rate of requests that generate responses with a 305 status code.
shown as request
gunicorn.request.status.307
(rate)
The rate of requests that generate responses with a 307 status code.
shown as request
gunicorn.request.status.308
(rate)
The rate of requests that generate responses with a 308 status code.
shown as request
gunicorn.request.status.400
(rate)
The rate of requests that generate responses with a 400 status code.
shown as request
gunicorn.request.status.401
(rate)
The rate of requests that generate responses with a 401 status code.
shown as request
gunicorn.request.status.402
(rate)
The rate of requests that generate responses with a 402 status code.
shown as request
gunicorn.request.status.403
(rate)
The rate of requests that generate responses with a 403 status code.
shown as request
gunicorn.request.status.404
(rate)
The rate of requests that generate responses with a 404 status code.
shown as request
gunicorn.request.status.405
(rate)
The rate of requests that generate responses with a 405 status code.
shown as request
gunicorn.request.status.406
(rate)
The rate of requests that generate responses with a 406 status code.
shown as request
gunicorn.request.status.407
(rate)
The rate of requests that generate responses with a 407 status code.
shown as request
gunicorn.request.status.408
(rate)
The rate of requests that generate responses with a 408 status code.
shown as request
gunicorn.request.status.409
(rate)
The rate of requests that generate responses with a 409 status code.
shown as request
gunicorn.request.status.410
(rate)
The rate of requests that generate responses with a 410 status code.
shown as request
gunicorn.request.status.411
(rate)
The rate of requests that generate responses with a 411 status code.
shown as request
gunicorn.request.status.412
(rate)
The rate of requests that generate responses with a 412 status code.
shown as request
gunicorn.request.status.413
(rate)
The rate of requests that generate responses with a 413 status code.
shown as request
gunicorn.request.status.414
(rate)
The rate of requests that generate responses with a 414 status code.
shown as request
gunicorn.request.status.415
(rate)
The rate of requests that generate responses with a 415 status code.
shown as request
gunicorn.request.status.416
(rate)
The rate of requests that generate responses with a 416 status code.
shown as request
gunicorn.request.status.417
(rate)
The rate of requests that generate responses with a 417 status code.
shown as request
gunicorn.request.status.419
(rate)
The rate of requests that generate responses with a 419 status code.
shown as request
gunicorn.request.status.421
(rate)
The rate of requests that generate responses with a 421 status code.
shown as request
gunicorn.request.status.422
(rate)
The rate of requests that generate responses with a 422 status code.
shown as request
gunicorn.request.status.423
(rate)
The rate of requests that generate responses with a 423 status code.
shown as request
gunicorn.request.status.424
(rate)
The rate of requests that generate responses with a 424 status code.
shown as request
gunicorn.request.status.426
(rate)
The rate of requests that generate responses with a 426 status code.
shown as request
gunicorn.request.status.428
(rate)
The rate of requests that generate responses with a 428 status code.
shown as request
gunicorn.request.status.429
(rate)
The rate of requests that generate responses with a 429 status code.
shown as request
gunicorn.request.status.431
(rate)
The rate of requests that generate responses with a 431 status code.
shown as request
gunicorn.request.status.451
(rate)
The rate of requests that generate responses with a 451 status code.
shown as request
gunicorn.request.status.500
(rate)
The rate of requests that generate responses with a 500 status code.
shown as request
gunicorn.request.status.501
(rate)
The rate of requests that generate responses with a 501 status code.
shown as request
gunicorn.request.status.502
(rate)
The rate of requests that generate responses with a 502 status code.
shown as request
gunicorn.request.status.503
(rate)
The rate of requests that generate responses with a 503 status code.
shown as request
gunicorn.request.status.504
(rate)
The rate of requests that generate responses with a 504 status code.
shown as request
gunicorn.request.status.505
(rate)
The rate of requests that generate responses with a 505 status code.
shown as request
gunicorn.request.status.506
(rate)
The rate of requests that generate responses with a 506 status code.
shown as request
gunicorn.request.status.507
(rate)
The rate of requests that generate responses with a 507 status code.
shown as request
gunicorn.request.status.508
(rate)
The rate of requests that generate responses with a 508 status code.
shown as request
gunicorn.request.status.510
(rate)
The rate of requests that generate responses with a 510 status code.
shown as request
gunicorn.request.status.511
(rate)
The rate of requests that generate responses with a 511 status code.
shown as request
gunicorn.request.status.512
(rate)
The rate of requests that generate responses with a 512 status code.
shown as request

Service Checks

gunicorn.is_running:

Returns CRITICAL if the Agent cannot find a Gunicorn master process, or if cannot find any working or idle worker processes.