Flat token buffer: lexer-fused parsing, flat mbe transcription, view-backed macro args, sink-replayed nonterminals [-1.14% incl tt-heavy, jemalloc, noise-controlled]

[xmakro]

The full flat-token architecture. Twelve commits:

1. Replace the tree-walking token cursor with a pre-flattened buffer. bump is an index increment, lookahead is direct indexing, parser snapshots and lazy capture replay share the buffer via refcount bumps, and whole delimited sequences skip in one step through an open/close match table. Skipped invisible delimiters stay in the buffer as entries (filtered on consumption) so tree-level lookahead and nesting-depth queries keep byte-exact semantics.
2. Emit the flat buffer directly from the lexer. Primary parses never construct a token tree at all; the few cold callers that need one (proc-macro from_str, cmdline attrs, diagnostics fake streams) rebuild it from the buffer. Delimiter-recovery diagnostics are preserved byte-for-byte.
3. Transcribe macro output directly into the flat buffer. mbe transcription writes through a FlatSink. Captured $x:tt fragments are shared at capture (a buffer-slice refcount bump, like the old cursor's Arc clone at parse_token_tree); splicing a captured fragment into expansion output copies its entries once per substitution (splice_slice extends the output buffer and rebases the match table). An earlier version of this description claimed splice-time sharing; that was wrong.
4. Pass macro arguments as lazy flat views. View-backed TokenStreams and bounded cursors let invocation arguments and expansion output parse straight from the buffer, removing the per-invocation flatten pass and the args tree rebuild (the two regression sources in the earlier prototype). Debug output is bounded to the viewed range and the pre-expansion KeywordIdents lint scans flat entries, so neither materializes every invocation in the crate. Attributes and macro-definition bodies stay eager so long-lived nodes do not pin token buffers.
5. Remove the now-dead tree-walking TokenCursor.
6. Share one Arc'd FlatBuffer between flat token cursors and slices. Cursor and slice clones are one refcount bump; both types are 16 bytes; LazyAttrTokenStreamInner drops from 88 to 64 bytes (vs main).
7. Make the flat tree rebuild helpers private to tokenstream.
8. Fix stable hashing, equality, and invariant guards in the flat token buffer. Length-prefixed stable hash, restored Arc pointer-equality fast paths, u32::MAX entry guard, enclosing_delimiter via the match table, assorted assertion hardening.
9. Fix flat lookahead scans at skipped invisible delimiters and hide buffer internals. The delimiter-counting scans walk from the raw cursor position so a leading skipped invisible open is counted, not entered; cursor/slice/sink fields are private; current_group_slice fails loudly in release builds on non-buffer-backed captures.
10. Add regression tests for the flat token buffer. Round-trip identity, bounded-view parsing at the view end, bounded Debug, view/eager equality, invisible-group tree lookahead.
11. Second review round, all findings fixed:
    • TokenStreamInner had grown to 48 bytes: a dead 32-byte OnceLock in every eager stream, paying for the rare view case. The Flat variant is now a niched Box; the enum is back at Vec size (24 bytes, static_assert_sized, so eager streams cost exactly what they did on main).
    • TokenStreamIter had regressed from slice::Iter to a per-element re-resolution of the stream backing (enum match + OnceLock check on every next()). It holds the materialized slice again; next() is a slice index.
    • The lexer hand-rolled the buffer's depth/match-table invariants as a third independent producer. It now builds through FlatSink (which gained patch_open_spacing for the after-the-fact open-delimiter spacing), and FlatBuffer, FlatEntry's fields, and FlatTokenCursor::from_parts are private: the sink is the only producer.
    • current_group_slice's token-equality debug assert compared spans that Parser::bump deliberately rewrites (dummy-span fixup); it compares kinds only now. The capture machinery's coupling to bump_with (injected tokens must not be open delimiters) is documented on bump_with.
    • The flat and tree KeywordIdents scans share one per-token helper so they cannot drift.
    • The /6 lexer presize and the finish slack threshold are backed by measured token density over rust-lang/rust itself (byte-weighted ~7.0 bytes/token for compiler/, ~5.9 for library/, per-file quartiles ~5.1/6.1/7.4, stress files down to ~1.7) with the undershoot/overshoot cost asymmetry spelled out in the comment.
    • metavar_expr_concat returns a Token, deleting an unreachable!.
    • New test: the lexer's buffer is a fixed point of rebuild-then-reflatten, entry-for-entry including depths, spacings, and the match table (flat_round_trip compares rebuilt trees, which cannot see depth or match divergence).
12. Replay captured nonterminal fragments straight into the transcription sink. A captured $e:expr/$t:ty/... is already a lazy replay over the flat buffer (LazyAttrTokenStreamInner::Pending); substituting it used to materialize that replay into an AttrTokenStream, convert to token trees, and re-flatten into the sink. FlatSink::splice_lazy now replays the capture token-by-token directly into the output. Fallbacks keep every hairy case on the tree path: attribute-bearing nodes, pending cfg/inner-attr replacements, broken last tokens, already-materialized streams, and captures starting with a metavar-invisible group (a conservative superset of the same-MetaVarKind unwrap rule). Delimiter balance is enforced with hard asserts, and under debug_assertions every fast-path splice re-runs the materializing path and asserts tree-for-tree equality — so the debug-assertions ui suite differentially validates the new path against the old one across every macro expansion it contains.

Known costs, stated plainly:

• A single buffer is capped at 2^32 entries; exceeding it is a compiler panic rather than silent truncation.
• Captured tt fragments and view-backed MacCall arguments pin their backing buffer; worst-case retention is (macro recursion depth) x (per-level buffer size) for tt-muncher-style recursion, since each level's output holds slices of the previous level's. The max-rss numbers below include the stress crates that hit this; worst measured cell is +1.8%.
• Splicing a tt fragment into expansion output copies it once per substitution; non-tt metavariables pay one splice_stream flatten of their from_ast stream per substitution.
• Proc-macro output still arrives as a tree and pays one flatten per expansion; this is the likely mechanism behind the token-stream-stress regression below, and bridge-side flat emission is the next lever. Proc-macro from_str and the other cold tree consumers pay a rebuild pass over the buffer.

Performance

Measured at the current head (all 12 commits) vs its base c9ff496: stage2 builds of both from the same worktree/config (CI LLVM, no PGO/BOLT), benched under jemalloc (LD_PRELOAD), instructions:u, Check+Debug, Full scenario, mbe/tt-heavy crate set. Noise floor established with a same-commit ThinLTO-control pair: base-vs-control geomean -0.000%, worst cell +/-0.036%, zero cells beyond +/-0.25%.

instructions:u geomean -1.141% (n=20), 12 cells improved >= 0.25%, 0 regressed >= 0.25%:

crate	check	debug
deep-vector	-7.75%	-7.39%
tt-muncher	-2.22%	-1.53%
clap_derive	-0.83%	-0.34%
syn	-0.54%	-0.17%
hyper	-0.53%	-0.27%
serde	-0.40%	-0.34%
ripgrep	-0.26%	-0.06%
derive	-0.06%	-0.04%
html5ever	+0.00%	+0.01%
token-stream-stress	+0.23%	+0.18%

The only cells above the noise floor in the wrong direction are token-stream-stress (+0.23%/+0.18%, both under the 0.25% significance threshold): it is the proc-macro pass-through stress, and proc-macro output still pays a per-expansion flatten (callgrind attributes the residual to bridge-side tree conversions; bridge-side flat emission is the named follow-up). The nonterminal sink-replay commit alone contributed -0.51% geomean with zero regressions, including deep-vector -4.4% and more than halving the token-stream-stress residual.

max-rss geomean -1.24%: deep-vector -14.7% check / -4.7% debug, serde -1.9/-1.2%, syn check -1.7%; worst increase tt-muncher debug +0.54%.

Numbers previously in this description (-1.07% geomean) were measured on glibc against a pre-rebase base without a control pair, and are withdrawn; the current numbers supersede them.

Correctness at the current head

• full tests/ui: 21635 passed / 0 failed; tests/incremental: 178 / 0; tests/pretty: 113 / 0
• rustc_parse (69) / rustc_ast / rustc_expand / rustc_lint unit tests green, including the six regression tests
• debug-assertions build: tests/ui 21648 passed / 0 failed, tests/incremental 178 / 0, all unit suites green — this run both exercises the commit-8/9 invariants (buffer well-formedness, level-alignment guards, splice preconditions; none fired) and differentially validates the sink-replay fast path against the tree path on every nonterminal substitution in the suite
• x check compiler green at every commit of the series

Remaining follow-ups (not blockers): bridge-side flat emission for proc-macro output (would remove the last sub-threshold residual); mbe attr/derive rules materialize a tree at the boundary; the commit stack should be squashed into its logical steps before an upstream submission.