Monitoring

Confluent Cloud Monitoring with the Metrics API

Confluent Cloud runs the brokers for you, which also means there is no JMX to scrape. Everything you know about your clusters comes from the Metrics API and your own clients. Here is how to get a complete picture anyway.

Published November 20, 2026 · 6 min read · KLogic Team

With a managed service the provider is responsible for broker health, patching and replication. You are still responsible for everything that affects your applications: throughput against your cluster's limits, consumer lag, client errors and cost. Confluent Cloud exposes that through the Metrics API, a time-series API separate from the Kafka protocol.

How the Metrics API works

  • It is served from api.telemetry.confluent.cloud and authenticated with a Cloud API key, not a cluster API key. Give the key's service account the minimum role it needs to read metrics.
  • The query endpoint answers ad-hoc questions: a metric, a time range, a granularity and optional grouping by labels such as topic or principal.
  • The export endpoint returns the latest values for a set of resources in Prometheus text format, so a Prometheus server can scrape it like any exporter.
  • Data arrives at one-minute granularity with a delay of a few minutes. That is fine for dashboards and most alerts, but it is not a real-time signal.

A minimal Prometheus scrape job for the export endpoint looks like this:

scrape_configs:
  - job_name: confluent-cloud
    scheme: https
    metrics_path: /v2/metrics/cloud/export
    params:
      resource.kafka.id: ["lkc-abc123"]
    basic_auth:
      username: <CLOUD_API_KEY>
      password: <CLOUD_API_SECRET>
    scrape_interval: 1m
    static_configs:
      - targets: ["api.telemetry.confluent.cloud"]

Scraping more often than once a minute adds load without adding information, and the API applies rate limits, so keep the interval at a minute or more.

The metrics that matter

MetricUse it for
received_bytes / sent_bytesIngress and egress throughput. Compare with your cluster's limits and use for cost forecasting.
received_records / sent_recordsRecord rates per topic. A sudden drop in received records is often the first sign of a producer outage.
retained_bytesStorage per topic. Watch growth and the topics that dominate it.
consumer_lag_offsetsLag per consumer group, topic and partition, measured by Confluent on the server side.
active_connection_count / request_countClient connections and request volume. Connection storms and request spikes show misbehaving clients.
partition_countPartitions against your cluster type's limit.
cluster_load_percentOn dedicated clusters, how busy the cluster is. Sustained high load means it is time to add capacity.

All of these are prefixed with io.confluent.kafka.server/ in API queries. Group by topic or consumer_group_id to break them down.

What you cannot see, and how to fill the gaps

The Metrics API covers the service side. Several things only your applications can tell you:

  • Client errors and retries. Collect producer record-error-rate and record-retry-rate, and consumer fetch latency, from your applications' own metrics.
  • Processing health. A consumer that commits offsets but writes bad data, or keeps rebalancing, looks fine in lag numbers until it is not. See our guide to consumer rebalancing.
  • Real-time lag. With a few minutes of delay, API lag is too slow for tight freshness targets. Measure lag directly through the Kafka admin API by comparing committed offsets with end offsets.
  • Broker internals. Request queues, ISR changes and JVM behaviour are Confluent's responsibility and are not exposed. Rely on their status page and support for those.

Querying the Metrics API directly

The query endpoint is useful for ad-hoc questions and for building reports. A request names one metric, an aggregation, a filter, a granularity and a time interval. For example, bytes received per topic over the last hour:

curl -s -u "$CLOUD_API_KEY:$CLOUD_API_SECRET" \
  -H "Content-Type: application/json" \
  https://api.telemetry.confluent.cloud/v2/metrics/cloud/query \
  -d '{
    "aggregations": [{ "metric": "io.confluent.kafka.server/received_bytes" }],
    "filter": { "field": "resource.kafka.id", "op": "EQ", "value": "lkc-abc123" },
    "granularity": "PT1M",
    "group_by": ["metric.topic"],
    "intervals": ["PT1H/now"]
  }'

The response is a list of timestamped values per topic. The same request shape works for consumer lag (group by metric.consumer_group_id) or for request counts by principal, which is a quick way to find the client responsible for a spike.

Using metrics for cost control

On a managed service, throughput and storage drive your bill. The Metrics API is the best source for understanding it:

  • Rank topics by retained_bytes to find where storage goes; often a few topics dominate.
  • Rank topics by ingress and egress to see which applications drive network cost.
  • Compare retained bytes with what consumers actually read; topics retained for weeks but only read in real time are candidates for shorter retention.

Our Kafka cost optimization guide covers the other levers, such as compression and partition counts.

Alerts worth setting up

  1. Consumer lag, converted to time, above each critical group's freshness target.
  2. Received records for a key topic dropping well below its normal level for the time of day.
  3. Throughput or partition count approaching your cluster type's limits.
  4. Retained bytes growing faster than planned, which drives both limits and cost.
  5. Active connections jumping suddenly, which often precedes throttling.

Mind the delay in alert rules. Because data arrives minutes late, alert "for" durations should be at least five minutes, and absent data should be treated as unknown rather than as zero, or you will get false alarms whenever the API is slow.

Monitoring Confluent Cloud alongside your other Kafka

Many organisations run Confluent Cloud next to self-hosted Kafka or AWS MSK. Separate tools for each make it hard to compare clusters or follow data across them. KLogic connects to Confluent alongside self-hosted and MSK clusters, so consumer lag, topic throughput and alerts look the same everywhere, and you can measure lag through the Kafka protocol rather than waiting for delayed metrics.

Alert examples in PromQL

Metrics from the export endpoint arrive in Prometheus with a confluent_kafka_server_ prefix and labels for the resource, topic and consumer group. Two illustrative rules show the shape:

# a critical topic stops receiving records
- alert: OrdersIngestStopped
  expr: sum(rate(confluent_kafka_server_received_records{topic="orders"}[10m])) == 0
  for: 10m

# lag above a fixed budget for an important group
- alert: CheckoutLagHigh
  expr: sum by (consumer_group_id) (confluent_kafka_server_consumer_lag_offsets{consumer_group_id="checkout"}) > 50000
  for: 15m

The long for durations absorb the API's delay. Offset thresholds are a starting point only; converting lag to time, as described in our SLO guide, gives alerts that match what users experience.

API key hygiene

  • Create a dedicated service account for monitoring, with read-only metrics access and nothing else.
  • Rotate the Cloud API key on a schedule and store it in your secret manager, not in Prometheus config files.
  • Use separate keys per environment so a leaked development key cannot read production metrics.

These habits matter because metrics reveal topic names and traffic patterns. Our security best practices cover access control in more depth.

Common questions

Can I get JMX metrics from Confluent Cloud?

No. Confluent operates the brokers and does not expose JMX. The Metrics API and your client applications' own metrics are the sources available to you.

How fresh is Confluent Cloud consumer lag data?

Metrics API data has one-minute granularity and arrives with a delay of a few minutes. For tighter freshness targets, measure lag through the Kafka admin API from your own tooling.

Can Prometheus scrape Confluent Cloud?

Yes, through the Metrics API export endpoint, which returns Prometheus text format. Keep the scrape interval at one minute or longer because of rate limits and the data granularity.

Which Confluent Cloud metrics should I alert on?

Consumer lag converted to time, drops in received records for critical topics, and usage approaching your cluster limits. See our guide to Kafka SLOs for alerting on lag without noise.

Can one tool monitor Confluent Cloud and self-hosted Kafka together?

Yes. Tools that connect through the Kafka protocol work with both. KLogic shows lag and throughput for Confluent, MSK and self-hosted clusters side by side.

See your Kafka clusters the way KLogic does

Consumer lag, replication health, alerts and AI-assisted debugging in one place, for self-hosted Kafka, AWS MSK and Confluent.

Free 14-day trial · No credit card required