You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[FEATURE] add Prometheus discovery and query tools - #135
Adds a read-only prometheus resource with five tools that discover and query the metrics of the Prometheus datasources of Perses, through the Perses datasource proxy:
Tool
Prometheus API
perses_list_prometheus_metric_names
/api/v1/label/__name__/values (+ /api/v1/metadata with include_metadata)
perses_list_prometheus_label_names
/api/v1/labels
perses_list_prometheus_label_values
/api/v1/label/<label>/values
perses_get_prometheus_metric_metadata
/api/v1/metadata
perses_query_prometheus
/api/v1/query, /api/v1/query_range
The calls use the Prometheus API client of client_golang (api/prometheus/v1) and the types of prometheus/common/model, with the HTTP client and default headers (credentials) of the Perses client. client_golang, prometheus/common and golang.org/x/sync were already indirect dependencies and become direct ones. github.com/perses/plugins/prometheus v0.59.1 is new: its Go SDK reads the plugin spec of the Prometheus datasources (go.mod and 2 lines in go.sum).
The results are compact JSON:
instant queries return the value of each series, highest first;
range queries return statistics per series (min, max, avg, last, minAt, maxAt, nonFinite), never raw points. min/max/avg leave out NaN and infinite samples (counted in nonFinite), last is the last sample, and series with only NaN/Inf samples are reported in a warning;
native histograms are rejected with a hint to use histogram_quantile(), histogram_count() or histogram_sum();
labels shared by all returned series are returned once in commonLabels, values keep 4 significant digits, results are capped (limit) and report seriesCount/returned.
Prometheus errors are returned unchanged (bad_data: invalid parameter "query": ... parse error ..., also on 503), other HTTP errors (e.g. from Perses) come with their status and message (HTTP 404 Not Found: document not found: ...), and a response over 1 MiB is an error that asks to aggregate the query. The POST → GET fallback of client_golang (403/405/501) is kept.
Metadata is looked up under the metric name first, then under its family name when the family type matches the suffix (_bucket/_count/_sum/_created of histograms and summaries, _total of counters). With include_metadata, metrics stay objects with a warning when metadata is unavailable.
datasource is optional. {name} / {name, project} select a datasource. {project} or no datasource selects the Prometheus datasource marked as default (project first, then global), or the only Prometheus datasource when none is marked as default. Otherwise the error lists the available Prometheus datasources. Resolutions are cached for 30 s; the comment notes that the cache assumes the single identity of the MCP server.
match takes one selector or a list (anyOf), since models often send a single string.
Invalid inputs are errors instead of silent fallbacks: invalid time_range, step or time_zone, time_range/step on an instant query, time on a range query.
Today an LLM can only reach Prometheus through perses_query_*_datasource. It has to know the Prometheus HTTP API, and it gets the raw response back. On a local Prometheus, a 6 h range query of rate(prometheus_http_requests_total[5m]) (61 series) is 241 KB through perses_query_global_datasource and about 3 KB with perses_query_prometheus (top 20 series, statistics). Large raw responses fill the context and are where models start to hallucinate values.
The discovery tools let the model check metric and label names instead of guessing them, which is the most common cause of failed PromQL. The same set of tools exists in the Grafana MCP server; here they go through Perses, so the datasource credentials stay in Perses and the Perses permissions apply.
perses_query_*_datasource stays available for the other endpoints and datasource kinds.
Tests
End-to-end tool tests with tooltest against a fake Perses (datasource API + proxy):
listing with search, limit, match (list and string), include_metadata (histogram base names, shortened help);
project datasources;
metadata (histogram suffixes, missing metrics);
instant, range and scalar queries (sorting, limit, common labels, NaN, rounding, time zones, step computation, absolute ranges);
errors (Prometheus 400 and 503 errors, Perses 400/404 messages, text error pages, native histograms, 1 MiB cap, wrong datasource kind, unknown datasource, invalid inputs) and the POST → GET fallback;
metadata lookups (exact name first, histogram and counter families with type check) and include_metadata when metadata is unavailable;
ordering by the unrounded maximum, NaN/Inf samples and overflow-safe averages.
The proxy requests carry the credentials of the Perses client (Authorization header).
The fake Perses embeds tooltest.Perses to record the requests.
Unit tests of the datasource resolution (defaults, single datasource, ambiguity, cache expiry), time ranges, steps, statistics and common labels.
Checked by hand against Perses + Prometheus: all tools, default and project datasources, Prometheus and Perses errors.
Adds a read-only prometheus resource for metric discovery and compact PromQL results through the existing Perses datasource proxy, complementing the generic datasource query tools.
Changes:
Introduces five tools for metric names, labels, label values, metadata, and queries.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The minimum-step rounding unnecessarily raises valid explicit subsecond steps to one second, discarding evaluation points and potentially changing reported extrema and averages.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Proxy URL construction sends all five tools to incorrect endpoints when the base URL contains a query string, and reported query timestamps lose fractional-second precision.
No open issues 🎉
2 previously missed issues
In code that hasn't changed since last review
Append proxy path to URL path before serialization
pkg/tools/prometheus/client.go:54
Appending the proxy path to the serialized URL breaks Perses base URLs containing a query string: with https://perses.example.com/perses?tenant=demo, the proxy path becomes part of the tenant value instead of the URL path. All five tools then target the wrong endpoint. Append the proxy path to the parsed URL's Path before serializing it.
Preserve fractional seconds in reported query timestamps
pkg/tools/prometheus/timerange.go:84
parseTime accepts fractional seconds and the query uses them, but time.RFC3339 drops them from the reported timestamps: a range from 10:00:00.100Z to 10:00:00.900Z reports identical bounds. The instant-query timestamp has the same issue in pkg/tools/prometheus/query.go:102. Use time.RFC3339Nano for both range bounds and the instant-query timestamp so the output preserves the times actually queried.
This might work for Prometheus but I wonder about the other data sources. Since perses uses a dynamic plugin architecture, would it be better that the perses MCP can act as a client for other MCP servers Like the Prometheus one, which already defines some of the functionallity that I think is included here? Basically leave the queries construction to specialized MCPs.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed to evaluate Prometheus metadata-matching and label-encoding edge cases identified in findings omitted from the published review.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed to judge whether the descriptions overstate Prometheus discovery guarantees, since approximate time bounds do not ensure samples exist within the requested interval.
Add the read-only prometheus resource with five tools that call the
Prometheus HTTP API of Perses datasources through the Perses proxy:
perses_list_prometheus_metric_names, perses_list_prometheus_label_names,
perses_list_prometheus_label_values, perses_get_prometheus_metric_metadata
and perses_query_prometheus.
They use the Prometheus API client of client_golang with the HTTP client
and the credentials of the Perses client, and return compact JSON: range
queries return statistics per series (min, max, avg, last, when min/max
happened) instead of raw points, labels shared by all series are returned
once, values keep 4 significant digits, responses are capped at 1 MiB and
Prometheus errors are returned unchanged.
The datasource is optional: the default Prometheus datasource of the
project, else the global one, is used, or the only Prometheus datasource
when none is marked as default.
Signed-off-by: Akshay Iyyadurai Balasundaram <akshay.iyyadurai.balasundaram@sap.com>
The Perses UI applies two settings of a Prometheus datasource itself, which
the Perses proxy doesn't. The tools now apply them as well, so that they
return the same data as the dashboards:
- queryParams (e.g. dedup or partial_response for Thanos) are added to every
request, unless the request has the parameter in its URL or its form body:
Prometheus merges a repeated parameter such as match[] from both. A value
with a dashboard variable is sent as is, with a warning.
- scrapeInterval (default 1m, as in the UI) is the minimum default step of
range queries, instead of 15s. An explicit step isn't raised to it.
The plugin spec is decoded with the Prometheus plugin SDK
(github.com/perses/plugins/prometheus/sdk/go/datasource). An empty scrape
interval, which Perses accepts and the UI saves when the field is cleared,
means the default one.
A datasource with a directUrl, only reachable by the browser, is an error
that explains it instead of an HTTP error of the proxy, also when it is the
default one. Without default, only the datasources that can be used are
candidates, and the others are reported with their own error. The default
datasource of a project is still returned before the global datasources are
listed.
Signed-off-by: Akshay Iyyadurai Balasundaram <akshay.iyyadurai.balasundaram@sap.com>
A list of metric names, label names or label values over 1 MiB returned the
advice for queries ("aggregate it (sum by, topk)"). It now asks to narrow
the list with match, search or time_range.
search was only applied once the whole list was downloaded, so it didn't
help with large lists. When the list is too large and search is set
without match, the list is requested again with search as a matcher, e.g.
{__name__=~"(?is).*memory.*"}, so that Prometheus filters it. The filter
isn't sent from the start as it may cost Prometheus more than a plain
listing. The local filter uses the same case-insensitive regular expression,
so that both return the same values, also for non-ASCII values.
Signed-off-by: Akshay Iyyadurai Balasundaram <akshay.iyyadurai.balasundaram@sap.com>
With include_metadata, the metadata of all the metrics is fetched in one
request, then most of it is dropped: on large instances it is over 1 MiB,
and the metrics were returned without metadata. The metadata of the
returned metrics is now looked up one by one in that case, at most 8 at a
time with errgroup, which cancels the lookups in progress on the first
error. The limit with include_metadata is capped at 200.
perses_get_prometheus_metric_metadata uses the same lookups.
Signed-off-by: Akshay Iyyadurai Balasundaram <akshay.iyyadurai.balasundaram@sap.com>
- The metadata of the _gcount and _gsum series of gauge histograms and of
the _info series of info metrics is looked up under their family name,
as Prometheus does. Gauge histograms have no _created series.
- perses_list_prometheus_label_values accepts the UTF-8 label names of
Prometheus 3, e.g. http.method. The name is escaped in the URL path
(U__http_2e_method) as Prometheus documents it, which also matches the
allowedEndpoints patterns of the Perses proxy, and quoted in the
selector of search.
- NaN is sorted after -Inf, so that a limited instant result keeps the
numbers first.
- Values keep all the digits of their integer part: with 4 significant
digits, a timestamp such as process_start_time_seconds was wrong by
days.
Signed-off-by: Akshay Iyyadurai Balasundaram <akshay.iyyadurai.balasundaram@sap.com>
Thanks @Nexucis! All the inline points are addressed (replies on each thread). On the rest of the review:
include_metadata fallback: capped in 5985fef. The max limit with include_metadata is now 50, the same as the default, so the fallback sends at most 100 requests (8 at a time). The docs and schema are updated, and TestMetadataLookupsAreBounded checks it.
PR description: updated to mention github.com/perses/plugins/prometheus.
Nits (e318d4e): slices.SortFunc + cmp.Compare in datasource.go. list now takes a listRequest struct. TestResolveStep, TestEscapeLabelName and TestRound use t.Run.
On Copilot's last review (same host, other port): 522afb6 restricted the followed redirects to the same host and port. 7315245 replaces these rules with a simpler one, as Nexucis first suggested: no redirect is followed. The Prometheus tools report a redirect as an error that names its location, and the proxy helper returns the 3xx response.
The datasource APIs don't redirect, and the requests carry the Perses credentials: a redirect is returned as the response by the proxy tools, and as an error that names its location by the Prometheus tools. This replaces the same host, port and scheme rules.
Signed-off-by: Akshay Iyyadurai Balasundaram <akshay.iyyadurai.balasundaram@sap.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a read-only
prometheusresource with five tools that discover and query the metrics of the Prometheus datasources of Perses, through the Perses datasource proxy:perses_list_prometheus_metric_names/api/v1/label/__name__/values(+/api/v1/metadatawithinclude_metadata)perses_list_prometheus_label_names/api/v1/labelsperses_list_prometheus_label_values/api/v1/label/<label>/valuesperses_get_prometheus_metric_metadata/api/v1/metadataperses_query_prometheus/api/v1/query,/api/v1/query_rangeclient_golang(api/prometheus/v1) and the types ofprometheus/common/model, with the HTTP client and default headers (credentials) of the Perses client.client_golang,prometheus/commonandgolang.org/x/syncwere already indirect dependencies and become direct ones.github.com/perses/plugins/prometheusv0.59.1 is new: its Go SDK reads the plugin spec of the Prometheus datasources (go.modand 2 lines ingo.sum).min,max,avg,last,minAt,maxAt,nonFinite), never raw points.min/max/avgleave out NaN and infinite samples (counted innonFinite),lastis the last sample, and series with only NaN/Inf samples are reported in a warning;histogram_quantile(),histogram_count()orhistogram_sum();commonLabels, values keep 4 significant digits, results are capped (limit) and reportseriesCount/returned.bad_data: invalid parameter "query": ... parse error ..., also on 503), other HTTP errors (e.g. from Perses) come with their status and message (HTTP 404 Not Found: document not found: ...), and a response over 1 MiB is an error that asks to aggregate the query. The POST → GET fallback ofclient_golang(403/405/501) is kept._bucket/_count/_sum/_createdof histograms and summaries,_totalof counters). Withinclude_metadata, metrics stay objects with a warning when metadata is unavailable.datasourceis optional.{name}/{name, project}select a datasource.{project}or no datasource selects the Prometheus datasource marked as default (project first, then global), or the only Prometheus datasource when none is marked as default. Otherwise the error lists the available Prometheus datasources. Resolutions are cached for 30 s; the comment notes that the cache assumes the single identity of the MCP server.matchtakes one selector or a list (anyOf), since models often send a single string.time_range,steportime_zone,time_range/stepon an instant query,timeon a range query.Docs:
docs/tools.md#prometheus,docs/overview.md,docs/configuration.md(resource list), README.Why
Today an LLM can only reach Prometheus through
perses_query_*_datasource. It has to know the Prometheus HTTP API, and it gets the raw response back. On a local Prometheus, a 6 h range query ofrate(prometheus_http_requests_total[5m])(61 series) is 241 KB throughperses_query_global_datasourceand about 3 KB withperses_query_prometheus(top 20 series, statistics). Large raw responses fill the context and are where models start to hallucinate values.The discovery tools let the model check metric and label names instead of guessing them, which is the most common cause of failed PromQL. The same set of tools exists in the Grafana MCP server; here they go through Perses, so the datasource credentials stay in Perses and the Perses permissions apply.
perses_query_*_datasourcestays available for the other endpoints and datasource kinds.Tests
tooltestagainst a fake Perses (datasource API + proxy):search,limit,match(list and string),include_metadata(histogram base names, shortened help);include_metadatawhen metadata is unavailable;Authorizationheader).tooltest.Persesto record the requests.