StackState-Go Expvar Integration

Overview

Use the StackState Expvar Agent check to:

  • Get information and monitor into your application memory usage
  • Instrument your own metrics
  • An example configration file for GO Expvar can be found here

Setup

Configuration

The Go Expvar integration requires the StackState Agent.

  1. Use Go’s expvar package to expose your memory information

    package ...
    
    import (
        ...
        "net/http"
        "expvar"
        ...
    )
    
    // If your application has no http server running for the DefaultServeMux,
    // you'll have to have a http server running for expvar to use, for example
    // by adding the following to your init function
    func init() {
        go http.ListenAndServe(":8080", nil)
    }
    
    ...
    
    // You can also expose variables that are specific to your application
    // See http://golang.org/pkg/expvar/ for more information
    
    var (
        exp_points_processed = expvar.NewInt("points_processed")
    )
    
    func processPoints(p RawPoints) {
        points_processed, err := parsePoints(p)
        exp_points_processed.Add(points_processed)
        ...
    }
    
    ...
    

  2. Configure the Agent to connect to your application’s expvar and specify the metrics you want to collect, edit conf.d/go_expvar.yaml

    init_config:
    instances:
       -   expvar_url: http://localhost:8080/debug/vars
           tags:
               - optionaltag1
               - optionaltag2
           metrics:
               - path: memstats/PauseTotalNs
                 alias: go_expvar.gc.pause_time_in_ns
                 type: rate                  # default is a gauge
               - path: memstats/Alloc        # will be reported as go_expvar.memstats.alloc
               - path: points_processed
                 type: rate
    

  3. Restart the Agent

Validation

Execute the info command and verify that the integration check has passed. The output of the command should contain a section similar to the following:

Checks
======

  [...]

  go_expvar
  ---------
      - instance #0 [OK]
      - Collected 8 metrics & 0 events

Data Collected

Metrics

go_expvar.memstats.alloc
(gauge)
Bytes allocated and not yet freed
shown as byte
go_expvar.memstats.frees
(gauge)
Number of frees
shown as operation
go_expvar.memstats.heap_alloc
(gauge)
Bytes allocated and not yet freed
shown as byte
go_expvar.memstats.heap_idle
(gauge)
Bytes in idle spans
shown as byte
go_expvar.memstats.heap_inuse
(gauge)
Bytes in non-idle spans
shown as byte
go_expvar.memstats.heap_objects
(gauge)
Total number of allocated objects
shown as item
go_expvar.memstats.heap_released
(gauge)
Bytes released to the OS
shown as byte
go_expvar.memstats.heap_sys
(gauge)
Bytes obtained from system
shown as byte
go_expvar.memstats.lookups
(gauge)
Number of pointer lookups
shown as operation
go_expvar.memstats.mallocs
(gauge)
Number of mallocs
shown as operation
go_expvar.memstats.num_gc
(gauge)
Number of garbage collections
shown as garbage collection
go_expvar.memstats.pause_ns.95percentile
(gauge)
95th percentile of recent GC pause durations
shown as nanosecond
go_expvar.memstats.pause_ns.avg
(gauge)
Average of recent GC pause durations
shown as nanosecond
go_expvar.memstats.pause_ns.count
(rate)
Number of submitted GC pause durations
shown as sample
go_expvar.memstats.pause_ns.max
(gauge)
Max GC pause duration
shown as nanosecond
go_expvar.memstats.pause_ns.median
(gauge)
Median GC pause duration
shown as nanosecond
go_expvar.memstats.pause_total_ns
(gauge)
Total GC pause duration over lifetime of process
shown as nanosecond
go_expvar.memstats.total_alloc
(gauge)
Bytes allocated (even if freed)
shown as byte