Skip to content

Support queue-level consumer metrics (diff / latency / consumer offset) #190

Open

Description

@xuanskyer

Is your feature request related to a problem? Please describe.

All consumer-side metrics are currently aggregated to broker granularity, so there is no way to tell which queue of a topic is lagging. When a single queue is stuck (a slow message, a stuck consumer instance, or an uneven rebalance), rocketmq_group_diff only shows that the group as a whole is behind — the skew between queues is invisible in Prometheus, and one has to fall back to the console (Topic -> Consumer detail) to see per-queue brokerOffset / consumerOffset / diff.

Interestingly, the label list for queue granularity already exists in the code but is never used:

https://github.com/apache/rocketmq-exporter/blob/master/src/main/java/org/apache/rocketmq/exporter/collector/RMQMetricsCollector.java#L479-L481

private static final List<String> GROUP_PULL_LATENCY_LABEL_NAMES = Arrays.asList(
    "cluster", "broker", "topic", "group", "queueid"
);

Describe the solution you'd like

The per-queue data is already fetched and iterated in MetricsCollectTask#collectConsumerOffset: consumeStats.getOffsetTable() is a Map<MessageQueue, OffsetWrapper>, and the per-queue lagTime is already computed one queue at a time. Both loops then fold the values into HashMap<String /* brokerName */, Long> and the queueId is dropped.

So exposing queue granularity needs no new admin call at all — queryMsgByOffset is already invoked once per queue, and the proposal keeps that unchanged. Suggested metrics:

metric meaning
rocketmq_queue_group_diff per-queue unconsumed messages (brokerOffset - consumerOffset)
rocketmq_queue_group_get_latency_by_storetime per-queue consume latency (ms)
rocketmq_queue_consumer_offset per-queue consumer offset

Cardinality concern and how it is addressed

Queue-level series multiply the existing consumer series by the number of queues per broker, which is not acceptable to enable globally on a large cluster. The proposal therefore keeps it off by default and gated by an explicit topic whitelist:

rocketmq:
  config:
    queueLevelTopics: "topic-a,topic-b"   # empty = disabled (default), "*" = all topics

Existing broker-level metrics are untouched, so current dashboards and alerts keep working.

Additional context

I have a working implementation (pure addition, no existing line modified, with unit tests for the whitelist parsing) and will open a PR referencing this issue.

Activity

  1. RockteMQ-AI commented on Sep 19, 2026

    @RockteMQ-AI

    Issue Evaluation

    Category: bug | Status: Under Review

    Thank you for reporting this issue. It has been flagged for code-level verification against the current codebase.

    A code engine analysis will be performed to verify the root cause. If confirmed, an automated fix proposal may be generated.


    Automated evaluation by RockteMQ-AI

  2. xuanskyer commented on Sep 20, 2026

    @xuanskyer
    Author

    Issue Evaluation

    Category: bug | Status: Under Review

    Thank you for reporting this issue. It has been flagged for code-level verification against the current codebase.

    A code engine analysis will be performed to verify the root cause. If confirmed, an automated fix proposal may be generated.

    Automated evaluation by RockteMQ-AI

    #191

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions