Skip to content

[FEATURE] add Prometheus discovery and query tools - #135

Merged
Nexucis merged 21 commits into
mainfrom
ibakshay-prometheus-tools
Nov 17, 2026
Merged

Nexucis merged 21 commits into
mainfrom
ibakshay-prometheus-tools

Conversation

@ibakshay

@ibakshay ibakshay commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

What

Adds a read-only prometheus resource with five tools that discover and query the metrics of the Prometheus datasources of Perses, through the Perses datasource proxy:

Tool Prometheus API
perses_list_prometheus_metric_names /api/v1/label/__name__/values (+ /api/v1/metadata with include_metadata)
perses_list_prometheus_label_names /api/v1/labels
perses_list_prometheus_label_values /api/v1/label/<label>/values
perses_get_prometheus_metric_metadata /api/v1/metadata
perses_query_prometheus /api/v1/query, /api/v1/query_range
  • The calls use the Prometheus API client of client_golang (api/prometheus/v1) and the types of prometheus/common/model, with the HTTP client and default headers (credentials) of the Perses client. client_golang, prometheus/common and golang.org/x/sync were already indirect dependencies and become direct ones. github.com/perses/plugins/prometheus v0.59.1 is new: its Go SDK reads the plugin spec of the Prometheus datasources (go.mod and 2 lines in go.sum).
  • The results are compact JSON:
    • instant queries return the value of each series, highest first;
    • range queries return statistics per series (min, max, avg, last, minAt, maxAt, nonFinite), never raw points. min/max/avg leave out NaN and infinite samples (counted in nonFinite), last is the last sample, and series with only NaN/Inf samples are reported in a warning;
    • native histograms are rejected with a hint to use histogram_quantile(), histogram_count() or histogram_sum();
    • labels shared by all returned series are returned once in commonLabels, values keep 4 significant digits, results are capped (limit) and report seriesCount/returned.
  • Prometheus errors are returned unchanged (bad_data: invalid parameter "query": ... parse error ..., also on 503), other HTTP errors (e.g. from Perses) come with their status and message (HTTP 404 Not Found: document not found: ...), and a response over 1 MiB is an error that asks to aggregate the query. The POST → GET fallback of client_golang (403/405/501) is kept.
  • Metadata is looked up under the metric name first, then under its family name when the family type matches the suffix (_bucket/_count/_sum/_created of histograms and summaries, _total of counters). With include_metadata, metrics stay objects with a warning when metadata is unavailable.
  • datasource is optional. {name} / {name, project} select a datasource. {project} or no datasource selects the Prometheus datasource marked as default (project first, then global), or the only Prometheus datasource when none is marked as default. Otherwise the error lists the available Prometheus datasources. Resolutions are cached for 30 s; the comment notes that the cache assumes the single identity of the MCP server.
  • match takes one selector or a list (anyOf), since models often send a single string.
  • Invalid inputs are errors instead of silent fallbacks: invalid time_range, step or time_zone, time_range/step on an instant query, time on a range query.

Docs: docs/tools.md#prometheus, docs/overview.md, docs/configuration.md (resource list), README.

Why

Today an LLM can only reach Prometheus through perses_query_*_datasource. It has to know the Prometheus HTTP API, and it gets the raw response back. On a local Prometheus, a 6 h range query of rate(prometheus_http_requests_total[5m]) (61 series) is 241 KB through perses_query_global_datasource and about 3 KB with perses_query_prometheus (top 20 series, statistics). Large raw responses fill the context and are where models start to hallucinate values.

The discovery tools let the model check metric and label names instead of guessing them, which is the most common cause of failed PromQL. The same set of tools exists in the Grafana MCP server; here they go through Perses, so the datasource credentials stay in Perses and the Perses permissions apply.

perses_query_*_datasource stays available for the other endpoints and datasource kinds.

Tests

  • End-to-end tool tests with tooltest against a fake Perses (datasource API + proxy):
    • listing with search, limit, match (list and string), include_metadata (histogram base names, shortened help);
    • project datasources;
    • metadata (histogram suffixes, missing metrics);
    • instant, range and scalar queries (sorting, limit, common labels, NaN, rounding, time zones, step computation, absolute ranges);
    • errors (Prometheus 400 and 503 errors, Perses 400/404 messages, text error pages, native histograms, 1 MiB cap, wrong datasource kind, unknown datasource, invalid inputs) and the POST → GET fallback;
    • metadata lookups (exact name first, histogram and counter families with type check) and include_metadata when metadata is unavailable;
    • ordering by the unrounded maximum, NaN/Inf samples and overflow-safe averages.
  • The proxy requests carry the credentials of the Perses client (Authorization header).
  • The fake Perses embeds tooltest.Perses to record the requests.
  • Unit tests of the datasource resolution (defaults, single datasource, ambiguity, cache expiry), time ranges, steps, statistics and common labels.
  • Checked by hand against Perses + Prometheus: all tools, default and project datasources, Prometheus and Perses errors.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The change still rejects valid UTF-8 label names, misses gauge-histogram metadata, and can rank NaN ahead of negative infinity in limited results.

3 open issues
What changed in this PR

Adds a read-only prometheus resource for metric discovery and compact PromQL results through the existing Perses datasource proxy, complementing the generic datasource query tools.

Changes:

  • Introduces five tools for metric names, labels, label values, metadata, and queries.
  • Adds cached datasource selection, range-query statistics, input validation, and response-size limits.
  • Registers the resource and adds documentation, unit tests, and tool-level tests.
File Description
README.md Introduces the Prometheus resource and tools.
pkg/​tools/​tool.go Adds the resource identifier and validation entry.
pkg/​tools/​prometheus/​timerange.go Resolves time ranges, query steps, and time zones.
pkg/​tools/​prometheus/​series.go Summarizes, sorts, and compacts query results.
pkg/​tools/​prometheus/​query.go Handles instant and range queries.
pkg/​tools/​prometheus/​prometheus.go Defines tool schemas, handlers, and read-only annotations.
pkg/​tools/​prometheus/​prometheus_test.go Tests tool behavior and proxy error handling.
pkg/​tools/​prometheus/​list.go Implements discovery and metadata lookup.
pkg/​tools/​prometheus/​internal_test.go Tests resolution, caching, timing, and statistics.
pkg/​tools/​prometheus/​helpers_test.go Provides fake Perses endpoints and test helpers.
pkg/​tools/​prometheus/​datasource.go Selects and caches Prometheus datasource references.
pkg/​tools/​prometheus/​client.go Adapts the API client to the authenticated Perses proxy.
internal/​permcp/​server.go Registers the Prometheus tools.
go.mod Promotes existing Prometheus dependencies to direct dependencies.
docs/​tools.md Documents parameters, results, and datasource selection.
docs/​overview.md Adds an overview of dedicated Prometheus capabilities.
docs/​configuration.md Lists the configurable Prometheus resource.

Review effort: Balanced

Comment thread pkg/tools/prometheus/list.go Resolved
Comment thread pkg/tools/prometheus/prometheus.go Resolved
Comment thread pkg/tools/prometheus/series.go Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The minimum-step rounding unnecessarily raises valid explicit subsecond steps to one second, discarding evaluation points and potentially changing reported extrema and averages.

2 open issues
2 resolved since last review

Review effort: Balanced

Comment thread pkg/tools/prometheus/timerange.go Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Proxy URL construction sends all five tools to incorrect endpoints when the base URL contains a query string, and reported query timestamps lose fractional-second precision.

No open issues 🎉

2 previously missed issues

In code that hasn't changed since last review

High severity Append proxy path to URL path before serialization

pkg/​tools/​prometheus/​client.go:54

Appending the proxy path to the serialized URL breaks Perses base URLs containing a query string: with https://perses.example.com/perses?tenant=demo, the proxy path becomes part of the tenant value instead of the URL path. All five tools then target the wrong endpoint. Append the proxy path to the parsed URL's Path before serializing it.

Medium severity Preserve fractional seconds in reported query timestamps

pkg/​tools/​prometheus/​timerange.go:84

parseTime accepts fractional seconds and the query uses them, but time.RFC3339 drops them from the reported timestamps: a range from 10:00:00.100Z to 10:00:00.900Z reports identical bounds. The instant-query timestamp has the same issue in pkg/tools/prometheus/query.go:102. Use time.RFC3339Nano for both range bounds and the instant-query timestamp so the output preserves the times actually queried.

2 resolved since last review

Review effort: Balanced

@jgbernalp

Copy link
Copy Markdown
Contributor

This might work for Prometheus but I wonder about the other data sources. Since perses uses a dynamic plugin architecture, would it be better that the perses MCP can act as a client for other MCP servers Like the Prometheus one, which already defines some of the functionallity that I think is included here? Basically leave the queries construction to specialized MCPs.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed to evaluate Prometheus metadata-matching and label-encoding edge cases identified in findings omitted from the published review.

No open issues 🎉

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues requiring further changes.

No open issues 🎉

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed to judge whether the descriptions overstate Prometheus discovery guarantees, since approximate time bounds do not ensure samples exist within the requested interval.

No open issues 🎉

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block approval of the read-only Prometheus tooling change.

No open issues 🎉

Review effort: Balanced

Add the read-only prometheus resource with five tools that call the
Prometheus HTTP API of Perses datasources through the Perses proxy:
perses_list_prometheus_metric_names, perses_list_prometheus_label_names,
perses_list_prometheus_label_values, perses_get_prometheus_metric_metadata
and perses_query_prometheus.

They use the Prometheus API client of client_golang with the HTTP client
and the credentials of the Perses client, and return compact JSON: range
queries return statistics per series (min, max, avg, last, when min/max
happened) instead of raw points, labels shared by all series are returned
once, values keep 4 significant digits, responses are capped at 1 MiB and
Prometheus errors are returned unchanged.

The datasource is optional: the default Prometheus datasource of the
project, else the global one, is used, or the only Prometheus datasource
when none is marked as default.

Signed-off-by: Akshay Iyyadurai Balasundaram <akshay.iyyadurai.balasundaram@sap.com>
The Perses UI applies two settings of a Prometheus datasource itself, which
the Perses proxy doesn't. The tools now apply them as well, so that they
return the same data as the dashboards:
- queryParams (e.g. dedup or partial_response for Thanos) are added to every
  request, unless the request has the parameter in its URL or its form body:
  Prometheus merges a repeated parameter such as match[] from both. A value
  with a dashboard variable is sent as is, with a warning.
- scrapeInterval (default 1m, as in the UI) is the minimum default step of
  range queries, instead of 15s. An explicit step isn't raised to it.

The plugin spec is decoded with the Prometheus plugin SDK
(github.com/perses/plugins/prometheus/sdk/go/datasource). An empty scrape
interval, which Perses accepts and the UI saves when the field is cleared,
means the default one.

A datasource with a directUrl, only reachable by the browser, is an error
that explains it instead of an HTTP error of the proxy, also when it is the
default one. Without default, only the datasources that can be used are
candidates, and the others are reported with their own error. The default
datasource of a project is still returned before the global datasources are
listed.

Signed-off-by: Akshay Iyyadurai Balasundaram <akshay.iyyadurai.balasundaram@sap.com>
A list of metric names, label names or label values over 1 MiB returned the
advice for queries ("aggregate it (sum by, topk)"). It now asks to narrow
the list with match, search or time_range.

search was only applied once the whole list was downloaded, so it didn't
help with large lists. When the list is too large and search is set
without match, the list is requested again with search as a matcher, e.g.
{__name__=~"(?is).*memory.*"}, so that Prometheus filters it. The filter
isn't sent from the start as it may cost Prometheus more than a plain
listing. The local filter uses the same case-insensitive regular expression,
so that both return the same values, also for non-ASCII values.

Signed-off-by: Akshay Iyyadurai Balasundaram <akshay.iyyadurai.balasundaram@sap.com>
With include_metadata, the metadata of all the metrics is fetched in one
request, then most of it is dropped: on large instances it is over 1 MiB,
and the metrics were returned without metadata. The metadata of the
returned metrics is now looked up one by one in that case, at most 8 at a
time with errgroup, which cancels the lookups in progress on the first
error. The limit with include_metadata is capped at 200.
perses_get_prometheus_metric_metadata uses the same lookups.

Signed-off-by: Akshay Iyyadurai Balasundaram <akshay.iyyadurai.balasundaram@sap.com>
- The metadata of the _gcount and _gsum series of gauge histograms and of
  the _info series of info metrics is looked up under their family name,
  as Prometheus does. Gauge histograms have no _created series.
- perses_list_prometheus_label_values accepts the UTF-8 label names of
  Prometheus 3, e.g. http.method. The name is escaped in the URL path
  (U__http_2e_method) as Prometheus documents it, which also matches the
  allowedEndpoints patterns of the Perses proxy, and quoted in the
  selector of search.
- NaN is sorted after -Inf, so that a limited instant result keeps the
  numbers first.
- Values keep all the digits of their integer part: with 4 significant
  digits, a timestamp such as process_start_time_seconds was wrong by
  days.

Signed-off-by: Akshay Iyyadurai Balasundaram <akshay.iyyadurai.balasundaram@sap.com>
Signed-off-by: Akshay Iyyadurai Balasundaram <akshay.iyyadurai.balasundaram@sap.com>
Signed-off-by: Akshay Iyyadurai Balasundaram <akshay.iyyadurai.balasundaram@sap.com>
Signed-off-by: Akshay Iyyadurai Balasundaram <akshay.iyyadurai.balasundaram@sap.com>
Signed-off-by: Akshay Iyyadurai Balasundaram <akshay.iyyadurai.balasundaram@sap.com>
Signed-off-by: Akshay Iyyadurai Balasundaram <akshay.iyyadurai.balasundaram@sap.com>
Signed-off-by: Akshay Iyyadurai Balasundaram <akshay.iyyadurai.balasundaram@sap.com>
@ibakshay

Copy link
Copy Markdown
Contributor Author

Thanks @Nexucis! All the inline points are addressed (replies on each thread). On the rest of the review:

  • include_metadata fallback: capped in 5985fef. The max limit with include_metadata is now 50, the same as the default, so the fallback sends at most 100 requests (8 at a time). The docs and schema are updated, and TestMetadataLookupsAreBounded checks it.
  • PR description: updated to mention github.com/perses/plugins/prometheus.
  • Nits (e318d4e): slices.SortFunc + cmp.Compare in datasource.go. list now takes a listRequest struct. TestResolveStep, TestEscapeLabelName and TestRound use t.Run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Redirect handling can leak Perses credentials to another host or transmit them in plaintext through an HTTPS-to-HTTP downgrade.

2 open issues

Review effort: Balanced

Comment thread pkg/tools/internal/proxy/proxy.go Resolved
Comment thread pkg/tools/prometheus/client.go Resolved
…ometheus error body

Signed-off-by: Akshay Iyyadurai Balasundaram <akshay.iyyadurai.balasundaram@sap.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human security assessment is needed because same-host redirects to a different port can forward Perses credentials to an unrelated service.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

Signed-off-by: Akshay Iyyadurai Balasundaram <akshay.iyyadurai.balasundaram@sap.com>
@ibakshay

ibakshay commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

On Copilot's last review (same host, other port): 522afb6 restricted the followed redirects to the same host and port. 7315245 replaces these rules with a simpler one, as Nexucis first suggested: no redirect is followed. The Prometheus tools report a redirect as an error that names its location, and the proxy helper returns the 3xx response.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The review records no active issues with the read-only Prometheus tooling change.

No open issues 🎉

Review effort: Balanced

The datasource APIs don't redirect, and the requests carry the Perses credentials: a redirect is returned as the response by the proxy tools, and as an error that names its location by the Prometheus tools. This replaces the same host, port and scheme rules.

Signed-off-by: Akshay Iyyadurai Balasundaram <akshay.iyyadurai.balasundaram@sap.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the new read-only Prometheus resource and tools.

No open issues 🎉

Review effort: Balanced

@Nexucis Nexucis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR is still in draft, but it looks good to me :)

@ibakshay
ibakshay marked this pull request as ready for review November 17, 2026 12:37
@Nexucis
Nexucis added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit 699a1c3 Nov 17, 2026
8 checks passed
@Nexucis
Nexucis deleted the ibakshay-prometheus-tools branch November 17, 2026 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants