StackState-Apache Kafka Consumer Integration

Overview

This Agent check only collects metrics for message offsets. If you want to collect metrics about the Kafka brokers themselves, see the kafka check.

This check fetches the highwater offsets from the Kafka brokers, consumer offsets for old-style consumers that store their offsets in zookeeper, and the calculated consumer lag (which is the difference between those two metrics).

Setup

Installation

This check is packaged with the Agent, so simply install the agent.

Configuration

Create a kafka_consumer.yaml file using this sample conf file as an example. Then restart the StackState Agent to start sending metrics to StackState.

Validation

Run the Agent’s info subcommand and look for kafka_consumer under the Checks section:

  Checks
  ======
    [...]

    kafka_consumer
    -------
      - instance #0 [OK]
      - Collected 26 metrics, 0 events & 1 service check

    [...]

Compatibility

The kafka_consumer check is compatible with all major platforms.

Data collected

Metrics

kafka.broker_offset
(gauge)
Current message offset on broker.
shown as offset
kafka.consumer_lag
(gauge)
Lag in messages between consumer and broker.
shown as offset
kafka.consumer_offset
(gauge)
Current message offset on consumer.
shown as offset

Events

The Kafka-consumer check does not include any event at this time.

Service Checks

The Kafka-consumer check does not include any service check at this time.

Troubleshooting

Specifying a non existent partition in your kafka_Consumer.yaml file

If you get this error in your info.log:

instance - #0 [Error]: ''

Specify the specific partition of your environment for your topic in your kafka_Consumer.yaml file:

#my_topic [0, 1, 4, 12]