feat(web): expose Prometheus metrics for the web process - #1570
Merged
Conversation
The web process had no instrumentation of any kind, so heap usage, GC duration, and event loop lag were invisible for the process that serves all application traffic. Diagnosing a recent GC-thrash incident required reading cgroup files inside the container and inferring the rest from trace spans. Mirror the backend's prom-client setup and serve it on its own port (WEB_METRICS_PORT, default 3070) rather than as a Next.js route, so scraping bypasses app middleware and is not reachable through the ingress. Also adds nodejs_heap_size_limit_bytes, which prom-client's default metrics omit. Without the ceiling, heap usage alone cannot distinguish a busy process from one pinned at its limit running back-to-back full mark-compacts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
Contributor
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
WalkthroughThe web package adds a Prometheus registry with Node.js and V8 heap metrics. A Node.js HTTP server exposes these metrics at ChangesPrometheus metrics
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant HTTPClient
participant MetricsServer
participant PrometheusRegistry
HTTPClient->>MetricsServer: GET /metrics
MetricsServer->>PrometheusRegistry: collect metrics
PrometheusRegistry-->>MetricsServer: Prometheus metric text
MetricsServer-->>HTTPClient: 200 response with metrics
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
8 tasks
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/web/src/metricsServer.ts`:
- Around line 15-18: Update the WEB_METRICS_PORT validation in the metrics
server startup flow to require an integer within the valid TCP port range,
including 65535 and excluding nonpositive values. Keep invalid values on the
existing logger.error and undefined-return path so server.listen is never called
with an out-of-range port.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: bd948e54-d265-43df-ba69-5ca840b4d8ad
⛔ Files ignored due to path filters (1)
yarn.lockis excluded by!**/yarn.lock,!**/*.lock
📒 Files selected for processing (7)
CHANGELOG.mdpackages/shared/src/env.server.tspackages/web/package.jsonpackages/web/src/instrumentation.tspackages/web/src/metricsServer.tspackages/web/src/promClient.test.tspackages/web/src/promClient.ts
Contributor
License Audit
Weak Copyleft Packages (informational)
Resolved Packages (8)
|
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit cf8db22. Configure here.
| if (!Number.isInteger(port) || port <= 0) { | ||
| logger.error(`Invalid WEB_METRICS_PORT '${env.WEB_METRICS_PORT}'; metrics server not started.`); | ||
| return undefined; | ||
| } |
There was a problem hiding this comment.
Incomplete metrics port validation
Medium Severity
The port guard only rejects non-integers and values <= 0, so ports above 65535 still reach server.listen. Node throws ERR_SOCKET_BAD_PORT synchronously there, which the 'error' listener does not catch. That exception escapes startMetricsServer and aborts register before initialize runs, so a bad WEB_METRICS_PORT can take down web startup instead of only skipping metrics.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit cf8db22. Configure here.
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The web process has no instrumentation of any kind. Heap usage, GC duration, and event loop lag are invisible for the process that serves every application request.
:3060is the backend worker and:6070is zoekt; both reportnodejs_*/go_*runtime metrics. The Next.js process reports nothing. Diagnosing a recent incident — a 6 s homepage render caused by V8 running back-to-back full mark-compacts at its heap ceiling — required reading/sys/fs/cgroupfiles inside the container and inferring the rest from trace span arithmetic, because the relevant numbers simply were not collected.Change
Mirror the backend's
prom-clientsetup for the web process:packages/web/src/promClient.tscollectDefaultMetrics+ a custom heap-limit gaugepackages/web/src/metricsServer.ts/metricson its own portpackages/web/src/instrumentation.tsnodejsruntimepackages/shared/src/env.server.tsWEB_METRICS_PORT, default3070This yields
nodejs_heap_size_used_bytes,nodejs_gc_duration_seconds,nodejs_eventloop_lag_p99_seconds,process_resident_memory_bytes, and the per-heap-space gauges.Two deliberate decisions:
A separate port, not a Next.js route. A
/api/metricsroute would pass through app middleware and be reachable through the ingress. A dedicated port keeps scraping off the request path and unexposed publicly, matching how the backend and zoekt already work.nodejs_heap_size_limit_bytesis added by hand, becausecollectDefaultMetricsomits it. This is the metric the recent investigation actually needed: without the ceiling, heap usage alone cannot distinguish a merely busy process from one pinned at its limit and thrashing.nodejs_heap_space_size_*can approximate it, but not legibly in an alert threshold.The metrics server is designed never to take down the web process — the
errorevent is handled rather than left to throw, and an unusable port is logged and skipped rather than passed tolisten(undefined), which would silently bind a random port and leave the scrape target quietly broken.Note for reviewers
prom-clientdoes not appear in.next/standalone/node_modules, which would normally mean aMODULE_NOT_FOUNDat startup. It is fully bundled instead: its own internal metric names are inlined into the server chunk and no externalrequire("prom-client")survives the build. Verified, but worth knowing if the bundling behaviour ever changes.Test plan
yarn workspace @sourcebot/web build— exit 0packages/web/src/promClient.test.ts— 4/4 passeslintclean on all new/changed filestsc --noEmitreports no errors in the new files (the 30 pre-existing errors are all in.test.tsfiles and unrelated)/metricsreturns 200 withtext/plain, containsnodejs_heap_size_limit_bytesandnodejs_heap_size_used_bytes, and other paths 404WEB_METRICS_PORTunset it logsInvalid WEB_METRICS_PORT 'undefined'; metrics server not started.and returns without bindingprom-clientis bundled into the standalone output rather than externalised:3070/metricsresponds in the cluster and the scrape lands in Better StackRequires
sourcebot-dev/sourcebot-infra#24 exposes port 3070 and adds the scrape. That PR also repairs the existing backend and zoekt scrapes, which turned out to have been failing silently since prod moved namespaces — no app metric has reached Better Stack in that window. Without it, these metrics are collected but never read.
🤖 Generated with Claude Code
Note
Low Risk
Additive observability only: a separate metrics server that is designed not to crash the web process, with no changes to auth, request handling, or data paths.
Overview
Adds Prometheus runtime metrics for the Next.js web process (heap, GC, event-loop lag), previously only available for the worker and Zoekt.
Metrics are served on a dedicated port (
WEB_METRICS_PORT, default3070) via a standalone HTTP server started frominstrumentation.ts, so scrapes bypass app middleware and stay off the public ingress. Includes a customnodejs_heap_size_limit_bytesgauge (omitted bycollectDefaultMetrics) so heap usage can be compared against V8's ceiling. Failures (bad port, listen errors) are logged without taking down the web process.Reviewed by Cursor Bugbot for commit cf8db22. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by CodeRabbit
New Features
/metricsendpoint on a configurable port, defaulting to3070.Documentation