feat(routing): session-routing plugin layer for per-session router affinity

[ajcasagrande]

Summary

One plugin category (session_routing) + one CLI flag (--session-routing <mode>, with repeatable --session-routing-opt key=value) covers every mechanism for telling an external router which session a request belongs to, so all of a conversation's turns re-land on the replica holding its KV prefix. "Session" (live instance keyed by x_correlation_id), not "conversation" (dataset template), is the affinity identity. Exactly one mode runs per invocation; the selected plugin is instantiated once per worker by InferenceClient and invoked at the request-serialization chokepoint.

Built-in modes

• dynamo_headers (preset) — X-Dynamo-Session-ID plus X-Dynamo-Parent-Session-ID on subagent children, for a Dynamo frontend running --router-session-affinity-ttl-secs.
• dynamo_nvext — nvext.session_control request-body metadata: bind (with timeout, option timeout_seconds, default 300) on every non-final turn, close on the final turn — for Dynamo builds that implement session_control. Plugin values win over dataset-shipped session_control keys; other nvext content is preserved.
• smg_routing_key (preset) — X-SMG-Routing-Key for the SGLang Model Gateway manual policy.
• session_id_header (preset) — additive X-Session-ID, parameterless.
• identity_headers — fully generic tiered identity headers emitting exactly what is configured, nothing by default. Options (each a comma-separable name list): session (this session's correlation ID), parent (immediate parent's ID, omitted for roots), root (session-tree root's ID — whole-tree affinity, e.g. pinning an entire agent tree to one replica for prefix-cache locality). Requires at least one name across tiers; names are validated as RFC 9110 tokens at config time and must be case-insensitively unique across all tiers combined. Each header preset is documented with its identity_headers equivalent.

Example commands

# Dynamo header affinity
aiperf profile --model my-model --url localhost:8000 --endpoint-type chat \
    --num-sessions 50 --session-turns-mean 4 --session-routing dynamo_headers

# Dynamo nvext session_control with a 600s bind timeout
aiperf profile --model my-model --url localhost:8000 --endpoint-type chat \
    --num-sessions 50 --session-turns-mean 4 \
    --session-routing dynamo_nvext --session-routing-opt timeout_seconds=600

# Layered routers + whole-tree affinity via the generic tiered mode
aiperf profile --model my-model --url localhost:8000 --endpoint-type chat \
    --num-sessions 50 --session-turns-mean 4 \
    --session-routing identity_headers \
    --session-routing-opt session=X-Session-ID,X-SMG-Routing-Key \
    --session-routing-opt root=X-Tree-ID

Removed (superseded)

--use-dynamo-conv-aware-routing / --use-dynamo-session-control, --use-legacy-dynamo-session-control, --dynamo-session-timeout-seconds, the dynamo_session_control module, AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID, AIPERF_HTTP_X_SMG_ROUTING_KEY_FROM_CORRELATION_ID, and the base_transports env-toggle header blocks.

Migration: --use-dynamo-conv-aware-routing → --session-routing dynamo_nvext; the SMG env toggle → --session-routing smg_routing_key; the X-Session-ID env toggle → --session-routing session_id_header.

Testing

• Six-mode wire-level raw-export integration matrix against the in-repo mock server (per-turn headers, bind/close lifecycle, stable session_id, header-modes-never-touch-the-body), plus live end-to-end runs of every mode.
• Binding unit tests for every terminal-path on_session_end notify (positive and negative), capability flags and skip semantics, plugin-contract init rejections, credit finality stamping at all TurnToSend seams, PAYLOAD_BYTES gates (preformat bail, format selection, cache-hit downgrade and its lookup-site wiring, cache-population skip), and the identity_headers tier/validation matrix.
• Full unit suite green apart from the 11 pre-existing base failures (weka loaders / mmap_cache, present on cquil11/aiperf-agentx-v1.0 before this branch).

🤖 Generated with Claude Code

[github-actions]

Try out this PR

Quick install:

pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@074a67d871eb3d7cd004ccde844cc2b34044dee8

Recommended with virtual environment (using uv):

uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@074a67d871eb3d7cd004ccde844cc2b34044dee8

---

Last updated for commit: 074a67d • Browse code

[ajcasagrande]

Added scope=lineage to dynamo_nvext (--session-routing-opt scope=lineage), incorporating the lineage-scoped affinity semantics from #14 (design and motivation by @YAMY1234, co-author credit on the commit).

Same key selection as #14 — every session in an agent tree binds the tree root's correlation ID — with one refinement to the close discipline: instead of closing on the root conversation's own final turn (which can tear down the shared key while sibling subagents are still running), close fires only on a request the credit issuer stamped is_tree_final (provably the last request of the whole tree, available under agentic replay); under indeterminate modes the session_control TTL reclaims the key, which is the same backstop #14 relies on for stragglers.

Verified with back-to-back --export-level raw mock-server runs of --scenario inferencex-agentx-mvp (with-subagents-060526 corpus) on both implementations:

run	child binds keyed on	closes observed
#14 --dynamo-session-affinity-scope lineage	tree-root ID (15/15)	0 (no root reached its final turn in the 60s window)
this PR scope=lineage	tree-root ID (54/54)	1 — on the provably-last wire request of a fully-drained tree
both, conversation scope (control)	own conversation ID	0

🤖 Generated with Claude Code

[ajcasagrande]

Converting to draft for now while I iron out a few details