atenet: preserve cold resume outcome and fallback empty template dimension by JeffLuoo · Pull Request #1482 · agent-substrate/substrate

Jeff Luo (JeffLuoo) · 2026-09-04T16:06:46Z

When atenet-router routes requests:

Failed cold resumes were mislabeled as ate.router.resume="none" due to an unconditional overwrite in ResumeActor. Preserves 'triggered' (for singleflight leader) and 'joined' (for joiners) on resume error, strictly reserving 'none' for warm hits on already-running actors.
Route duration metrics emitted empty strings for ate.template.atespace and ate.template.name when routing/resuming failed, violating the required metric registry contract. Normalizes empty template namespace and name to "unknown", avoiding request-derived/caller-controlled metric labels on failure paths.
Updated metric registry documentation for ate.template.atespace and ate.template.name to include "unknown" fallback.

It's a good idea to open an issue first for discussion.

Tests pass
Appropriate changes to documentation are included in the PR

Krisztian F (krisztianfekete) · 2026-09-04T17:36:01Z

+// Note: "unknown" is syntactically a legal atespace/template name; like
+// ate.sandbox.class="unknown", this trades potential collision with a real
+// object named "unknown" for maintaining a bounded, non-empty metric dimension.
+const TemplateUnknown = "unknown"


Can we have a Normalize for this, like for the rest of unknowns?

Added normalization in ateattr, following the pattern of NormalizeSandboxClass and NormalizeOperationName, and switched recordRouteDuration to use it. Added unit tests for it in ateattr_test.go as well.

Krisztian F (krisztianfekete) · 2026-09-04T17:43:06Z

+		// - ResumeOutcomeJoined ("joined"): Cold activation joiner (resumed == true or attempt errored, caller's reqID != leaderID).
 		outcome := ResumeOutcomeNone
-		if callRes.resumed {
+		if callRes.resumed || callRes.err != nil {


A NotFound never triggered anything, but it now gets resume="triggered". Can we add a new value like unattempted for definitive errors, and keep triggered/joined for the capacity ones where a resume really was in flight?

Great catch. Added RouterResumeUnattempted = "unattempted" to ateattr and registered it in metrics.yaml.

Krisztian F (krisztianfekete) · 2026-09-04T17:43:47Z

-		// - ResumeOutcomeNone ("none"): resumed == false, actor was already active/running.
-		// - ResumeOutcomeTriggered ("triggered"): Cold activation leader (resumed == true, caller's reqID == leaderID).
-		// - ResumeOutcomeJoined ("joined"): Cold activation joiner (resumed == true, caller's reqID != leaderID).
+		// - ResumeOutcomeNone ("none"): resumed == false and err == nil, actor was already active/running.


Here, + the PR says it reserves none for warm hits, but we have multiple places it still returns none on a cancel.

Updated. Set none only when actual warm hits where the actor was already running.

Krisztian F (krisztianfekete) · 2026-09-04T17:48:09Z

+          The atespace of the ActorTemplate of the actor, or "unknown" if the
+          request failed before template resolution.


Two things:

"if the request failed" misses egress. Egress succeeds and still has no template, so it gets unknown too. Result's comment in extproc/handler.go phrases it right, can we mirror that?

7 metrics share this brief but only the router sends unknown. And line 72 plus bounded-or-catalog-scoped in docs/metrics/substrate.yaml both say these labels only ever name something an operator made. unknown doesn't, so it needs writing down there.

Updated both.

Krisztian F (krisztianfekete) · 2026-09-04T17:49:41Z

 		}
-		if outcome != ResumeOutcomeNone {
-			t.Errorf("expected outcome %q on error, got %q", ResumeOutcomeNone, outcome)
+		if outcome != ResumeOutcomeTriggered {


Does this expect a nonexistent actor trigering a cold activation basically?

Fixed. With unattempted introduced, ActorNotFound now expects ResumeOutcomeUnattempted (since no cold activation was ever attempted). Also added a test case for context cancellation verifying it returns unattempted.

…nsions (agent-substrate#1474) Fixes agent-substrate#1474 When atenet-router routes requests: - Mislabeled cold resume failures: Previously, failed resumes were unconditionally overwritten with ate.router.resume="none", distorting cold activation rates and durations. In-flight cold activation attempts (and joiners) now preserve "triggered" and "joined" respectively on capacity and transient errors. Definitive non-activation errors (e.g. NotFound, InvalidArgument), context cancellations, and non-resuming directions (egress) now report "unattempted", strictly reserving "none" for warm hits on running actors. - Missing template labels: Empty template namespace and name dimensions are normalized to "unknown" using ateattr.NormalizeTemplateDimension, satisfying metric registry invariants without using request-derived labels on failure paths. - Documentation: Updated registry metrics.yaml and substrate.yaml to reflect the "unattempted" resume value, mirror handler Result documentation for "unknown" template fallback (including egress), and account for platform-injected unknown dimensions.

Krisztian F (krisztianfekete) · 2026-09-07T08:46:35Z

+
+// NormalizeTemplateDimension ensures a template dimension (atespace or name) is
+// non-empty, falling back to TemplateUnknown if unset.
+func NormalizeTemplateDimension(dim string) string {


Nice, but it's still only called from the router though, and the group brief now promises unknown for all 7 metrics that ref these attrs.

Krisztian F (krisztianfekete) · 2026-09-07T08:49:28Z

+// isDefinitiveResumeError reports whether err represents a failure where no cold
+// activation could be attempted (e.g. the actor does not exist, bad request, or
+// permission denied), as opposed to in-flight capacity or transient failures.
+func isDefinitiveResumeError(err error) bool {
+	switch status.Code(err) {
+	case codes.NotFound, codes.InvalidArgument, codes.PermissionDenied, codes.Unauthenticated:
+		return true
+	default:
+		return false
+	}
+}


Can we flip the default? Right now Internal, Unimplemented, Unknown etc. fall through to triggered, so an unrecognized code is as an activation in the latency series.
Unknown means we don't know, so unattempted seems safer.

Also wondering about FailedPrecondition, its registry brief is "the state of the actor did not permit a route" and with parking off it fails immediately, which sounds definitive. And retryable's comment just above already groups DeadlineExceeded with NotFound and PermissionDenied, so it's a bit weird to have two classifiers for the same codes I think.

Wdyt?

Krisztian F (krisztianfekete) · 2026-09-07T08:52:04Z

 	// RouterResumeTriggered indicates this request won the singleflight lock and initiated cold activation.
 	RouterResumeTriggered = "triggered"


initiated cold activation

a request that is ResourceExhausted gets triggered without activating anything. Same wording in the registry brief.

Krisztian F (krisztianfekete) · 2026-09-07T09:01:31Z

+                No cold activation was in flight. The request was invalid, the
+                actor did not exist, the direction does not resume, or the
+                request canceled before activation was attempted.


"No cold activation was in flight" and "request canceled" contradict each other, a canceled joiner was waiting on someone else's activation. Maybe "this request neither attempted nor observed an activation" is more accurate?

Jeff Luo (JeffLuoo) requested review from Da Huang (git286) and Krisztian F (krisztianfekete) September 4, 2026 16:06

Krisztian F (krisztianfekete) reviewed Sep 4, 2026

View reviewed changes

Jeff Luo (JeffLuoo) force-pushed the fix/issue-1474-router-route-duration-labels branch from 1cbcde7 to 9546f66 Compare September 4, 2026 18:19

Jeff Luo (JeffLuoo) requested a review from Krisztian F (krisztianfekete) September 4, 2026 18:23

Krisztian F (krisztianfekete) reviewed Sep 7, 2026

View reviewed changes

		The atespace of the ActorTemplate of the actor, or "unknown" if the
		request failed before template resolution.

		// RouterResumeTriggered indicates this request won the singleflight lock and initiated cold activation.
		RouterResumeTriggered = "triggered"

Uh oh!

Conversation

Jeff Luo (JeffLuoo) commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Jeff Luo (JeffLuoo) commented Sep 4, 2026 •

edited

Loading