Cache derive proc macro expansion with incremental query by Kobzol · Pull Request #145354 · rust-lang/rust

Kobzol · 2025-08-13T13:09:12Z

This is a revival of #129102, originally implemented by @futile. Since it looks like they are not active currently, I'd like to push this work forward.

The first commit is squashed and rebased work from the original PR, with author attribution to futile. The rest of the commits are some additional comments that I created mostly for myself to understand what happens here. I also did some cleanups based on Vadim's review comments on the original PR, plus I refactored the TLS access a bit using scoped_tls.

The biggest issue, as usually, are tests... I tried using #[rustc_clean(..., loaded_from_disk = "derive_macro_expansion")], but the problem is that since this query cannot recover the original key from its hash, and thus its fingerprintstyle is FingerprintStyle::Opaque, this crashes when I try to use loaded_from_disk. Any suggestions from someone who actually understands the query system would be welcome 😅

You can try it yourself on nightly with RUSTFLAGS="-Zcache-proc-macros" cargo +nightly build.

On a no-op change re-check of octocrab 0.49 (which has a ton of serde derive proc macro invocations), this saves ~0.6s out of ~6s (so a ~10% win) on my PC.

Tracking issue: #151364

r? @petrochenkov

rustbot · 2025-08-13T13:09:17Z

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

Kobzol · 2025-08-13T13:09:30Z

@bors try @rust-timer queue

Cache derive proc macro expansion with incremental query

Kobzol · 2025-08-13T14:56:39Z

@bors try

Cache derive proc macro expansion with incremental query

Kobzol · 2025-08-15T11:01:20Z

I looked into the benchmark failure on diesel, seems like I hold queries wrong somehow :) Any suggestions on what could be the case? Maybe the derived Hash impl for TokenStream does not correspond to the manual Eq impl? 🤔

thread 'rustc' panicked at /projects/personal/rust/rust/compiler/rustc_query_system/src/query/plumbing.rs:191:23:
active query job entry
stack backtrace:
   0: __rustc::rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::option::expect_failed
   3: <rustc_query_system::query::plumbing::JobOwner<(rustc_span::hygiene::LocalExpnId, rustc_data_structures::svh::Svh, &rustc_ast::tokenstream::TokenStream), rustc_query_system::query::QueryStackDeferred>>::complete::<rustc_query_system::query::caches::DefaultCache<(rustc_span::hygiene::LocalExpnId, rustc_data_structures::svh::Svh, &rustc_ast::tokenstream::TokenStream), rustc_middle::query::erase::Erased<[u8; 8]>>>
   4: rustc_query_system::query::plumbing::try_execute_query::<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::DefaultCache<(rustc_span::hygiene::LocalExpnId, rustc_data_structures::svh::Svh, &rustc_ast::tokenstream::TokenStream), rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt, true>
   5: <scoped_tls::ScopedKey<rustc_expand::proc_macro::QueryDeriveExpandCtx>>::set::<<rustc_expand::proc_macro::QueryDeriveExpandCtx>::enter<<rustc_expand::proc_macro::DeriveProcMacro as rustc_expand::base::MultiItemModifier>::expand::{closure#1}::{closure#0}, core::result::Result<rustc_ast::tokenstream::TokenStream, ()>>::{closure#0}, core::result::Result<rustc_ast::tokenstream::TokenStream, ()>>
   6: <rustc_expand::proc_macro::QueryDeriveExpandCtx>::enter::<<rustc_expand::proc_macro::DeriveProcMacro as rustc_expand::base::MultiItemModifier>::expand::{closure#1}::{closure#0}, core::result::Result<rustc_ast::tokenstream::TokenStream, ()>>
   7: <rustc_expand::proc_macro::DeriveProcMacro as rustc_expand::base::MultiItemModifier>::expand
   8: <rustc_expand::expand::MacroExpander>::expand_invoc
   9: <rustc_expand::expand::MacroExpander>::fully_expand_fragment
  10: <rustc_expand::expand::MacroExpander>::expand_crate
  11: <rustc_session::session::Session>::time::<rustc_ast::ast::Crate, rustc_interface::passes::configure_and_expand::{closure#1}>
  12: rustc_interface::passes::resolver_for_lowering_raw
      [... omitted 6 frames ...]
  13: <rustc_middle::ty::context::TyCtxt>::resolver_for_lowering
  14: <std::thread::local::LocalKey<core::cell::Cell<*const ()>>>::with::<rustc_middle::ty::context::tls::enter_context<<rustc_middle::ty::context::GlobalCtxt>::enter<rustc_interface::passes::create_and_enter_global_ctxt<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>::{closure#2}::{closure#0}, core::option::Option<rustc_interface::queries::Linker>>::{closure#1}, core::option::Option<rustc_interface::queries::Linker>>::{closure#0}, core::option::Option<rustc_interface::queries::Linker>>
  15: <rustc_middle::ty::context::TyCtxt>::create_global_ctxt::<core::option::Option<rustc_interface::queries::Linker>, rustc_interface::passes::create_and_enter_global_ctxt<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>::{closure#2}::{closure#0}>
  16: <rustc_interface::passes::create_and_enter_global_ctxt<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>::{closure#2} as core::ops::function::FnOnce<(&rustc_session::session::Session, rustc_middle::ty::context::CurrentGcx, alloc::sync::Arc<rustc_data_structures::jobserver::Proxy>, &std::sync::once_lock::OnceLock<rustc_middle::ty::context::GlobalCtxt>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_middle::arena::Arena>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_hir::Arena>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2})>>::call_once::{shim:vtable#0}
  17: <alloc::boxed::Box<dyn for<'a> core::ops::function::FnOnce<(&'a rustc_session::session::Session, rustc_middle::ty::context::CurrentGcx, alloc::sync::Arc<rustc_data_structures::jobserver::Proxy>, &'a std::sync::once_lock::OnceLock<rustc_middle::ty::context::GlobalCtxt<'a>>, &'a rustc_data_structures::sync::worker_local::WorkerLocal<rustc_middle::arena::Arena<'a>>, &'a rustc_data_structures::sync::worker_local::WorkerLocal<rustc_hir::Arena<'a>>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}), Output = core::option::Option<rustc_interface::queries::Linker>>> as core::ops::function::FnOnce<(&rustc_session::session::Session, rustc_middle::ty::context::CurrentGcx, alloc::sync::Arc<rustc_data_structures::jobserver::Proxy>, &std::sync::once_lock::OnceLock<rustc_middle::ty::context::GlobalCtxt>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_middle::arena::Arena>, &rustc_data_structures::sync::worker_local::WorkerLocal<rustc_hir::Arena>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2})>>::call_once
  18: rustc_interface::passes::create_and_enter_global_ctxt::<core::option::Option<rustc_interface::queries::Linker>, rustc_driver_impl::run_compiler::{closure#0}::{closure#2}>
  19: std::panic::catch_unwind::<core::panic::unwind_safe::AssertUnwindSafe<rustc_interface::interface::run_compiler<(), rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}::{closure#0}>, ()>
  20: <scoped_tls::ScopedKey<rustc_span::SessionGlobals>>::set::<rustc_interface::util::run_in_thread_with_globals<rustc_interface::util::run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<(), rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}::{closure#0}, ()>
  21: rustc_span::create_session_globals_then::<(), rustc_interface::util::run_in_thread_with_globals<rustc_interface::util::run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<(), rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}::{closure#0}>
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace

Kobzol · 2025-08-25T14:54:25Z

Ok I did a bit of debugging, and found a fun fact: TokenStream == TokenStream is (or at least can be) false. So that has to be fixed before we can use token streams as query keys.. :)

Kobzol · 2025-08-25T14:57:04Z

Ok I found it, it's this. @nnethercote Do you think it's possible to make TokenStreams comparable?

petrochenkov · 2025-08-25T17:42:02Z

Technically, removing that PartialEq for InvisibleOrigin is a language change.
But this PR can be unblocked without the lang team involvement by migrating the macro_rules matching logic specifically from PartialEq to a custom comparison function.

I'm not actually sure where else PartialEq for TokenStream is used, besides the matching and now also queries.
UPD: Nowhere.
UPD2: I think (Partial)Eq for TokenStream can also be derived now.

Kobzol · 2025-08-25T18:48:25Z

Could you please specify what do you mean by "macro rules matching logic"? I tried removing PartialEq from InvisibleOrigin, which required removing it from Delimiter and TokenKind, and that makes a bunch of places in code not compile, because they use == on these types (approx. 13 usages across 4 files). All of that should be migrated to a custom trait?

I also thought that another alternative would be to wrap TokenStream in a newtype and implement Eq for that newtype, which would delegate to a custom comparison trait for the query key logic (so basically an inverse of what you proposed).

Kobzol · 2026-01-15T15:37:58Z

@rustbot ready

petrochenkov · 2026-01-15T15:47:45Z

r=me after addressing the remaining nits and squashing commits.
@rustbot author

Kobzol · 2026-01-16T08:30:14Z

@bors r=petrochenkov

rust-bors · 2026-01-16T08:30:16Z

📌 Commit 8fa2f69 has been approved by petrochenkov

It is now in the queue for this repository.

@futile

…henkov Cache derive proc macro expansion with incremental query This is a revival of rust-lang#129102, originally implemented by @futile. Since it looks like they are not active currently, I'd like to push this work forward. The first commit is squashed and rebased work from the original PR, with author attribution to futile. The rest of the commits are some additional comments that I created mostly for myself to understand what happens here. I also did some cleanups based on Vadim's review comments on the original PR, plus I refactored the TLS access a bit using `scoped_tls`. The biggest issue, as usually, are tests... I tried using `#[rustc_clean(..., loaded_from_disk = "derive_macro_expansion")]`, but the problem is that since this query cannot recover the original key from its hash, and thus its fingerprintstyle is `FingerprintStyle::Opaque`, [this](https://github.com/rust-lang/rust/blob/2fef0a30ae6b2687dfb286cb544d2a542f7e2335/compiler/rustc_incremental/src/persist/dirty_clean.rs#L388) crashes when I try to use `loaded_from_disk`. Any suggestions from someone who actually understands the query system would be welcome 😅 TODO: document the new unstable flag On a no-op change re-check of `octocrab 0.49` (which has a ton of `serde` derive proc macro invocations), this saves ~0.6s out of ~6s (so a ~10% win) on my PC. r? @petrochenkov

@ghost

Rollup of 6 pull requests Successful merges: - #145354 (Cache derive proc macro expansion with incremental query) - #151123 (Support primitives in type info reflection) - #151178 (simplify words initialization using Rc::new_zeroed) - #151187 (Use `default_field_values` more in `Resolver`) - #151197 (remove lcnr from compiler review rotation) - #151203 (Revert `QueryStackFrame` split) r? @ghost

@futile

Rollup merge of #145354 - cache-proc-derive-macros, r=petrochenkov Cache derive proc macro expansion with incremental query This is a revival of #129102, originally implemented by @futile. Since it looks like they are not active currently, I'd like to push this work forward. The first commit is squashed and rebased work from the original PR, with author attribution to futile. The rest of the commits are some additional comments that I created mostly for myself to understand what happens here. I also did some cleanups based on Vadim's review comments on the original PR, plus I refactored the TLS access a bit using `scoped_tls`. The biggest issue, as usually, are tests... I tried using `#[rustc_clean(..., loaded_from_disk = "derive_macro_expansion")]`, but the problem is that since this query cannot recover the original key from its hash, and thus its fingerprintstyle is `FingerprintStyle::Opaque`, [this](https://github.com/rust-lang/rust/blob/2fef0a30ae6b2687dfb286cb544d2a542f7e2335/compiler/rustc_incremental/src/persist/dirty_clean.rs#L388) crashes when I try to use `loaded_from_disk`. Any suggestions from someone who actually understands the query system would be welcome 😅 TODO: document the new unstable flag On a no-op change re-check of `octocrab 0.49` (which has a ton of `serde` derive proc macro invocations), this saves ~0.6s out of ~6s (so a ~10% win) on my PC. r? @petrochenkov

futile · 2026-01-16T20:27:24Z

Congratz @Kobzol, and thanks for seeing this through! The final code looks super nice. And thanks to @petrochenkov for all the reviews :) Looking forward to updating to the next nightly 😀

… r=jackh726 Document `-Zcache-proc-macros` Forgot about this in rust-lang#145354. Created a tracking issue in rust-lang#151364.

Rollup merge of #151227 - Kobzol:document-cache-proc-macros, r=jackh726 Document `-Zcache-proc-macros` Forgot about this in #145354. Created a tracking issue in #151364.

…4595) # Objective Add flag to improve incremental build times. ## Solution Add the flag `-Zcache-proc-macros` to config_fast_build_times.config so that a user can easily enable it if they want to. ## Testing I did a few tests back and forth with and without this on Linux. I saw consistent incremental build time gains, maybe around 5%. I've also been running with this on aarch64 Mac so it compiles there as well. I haven't tested this on Windows. I unfortunately did not save the run results I gathered. --- Another thing to note is that this might _slightly_ increase clean compilation times but only by a small fraction (maybe a couple of percent, not sure). I'm guessing this might be a hot path in the compiler so branching on this condition every time could possibly make it slightly slower? Or it might have just been noise. The claim that it improves incremental by 5-10% is based on my own experiments and the original rust [PR](rust-lang/rust#145354) that states > On a no-op change re-check of octocrab 0.49 (which has a ton of serde derive proc macro invocations), this saves ~0.6s out of ~6s (so a ~10% win) on my PC. Maybe we shouldn't give a number at all? It might even be the case that a user doesn't notice this a lot of the time. My suspicion is that this would make more of a difference if a crate has many and large derive macros, relative to the rest of the code size. Also I asked about this in the Discord server, but this will probably return stale results if a proc macro if there are side effects. Does Bevy have any side effects in proc macros?

rustbot assigned petrochenkov Aug 13, 2025

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 13, 2025

rustbot added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Aug 13, 2025

This comment has been minimized.

Sign in to view

rust-bors Bot added a commit that referenced this pull request Aug 13, 2025

Auto merge of #145354 - Kobzol:cache-proc-derive-macros, r=<try>

21e746d

Cache derive proc macro expansion with incremental query

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Aug 13, 2025

petrochenkov mentioned this pull request Aug 13, 2025

Experimental: Add Derive Proc-Macro Caching #129102

Closed

3 tasks

bjorn3 reviewed Aug 13, 2025

View reviewed changes

Comment thread compiler/rustc_ast/src/tokenstream.rs Outdated

bjorn3 reviewed Aug 13, 2025

View reviewed changes

Comment thread compiler/rustc_middle/src/query/mod.rs Outdated

bjorn3 reviewed Aug 13, 2025

View reviewed changes

Comment thread compiler/rustc_expand/src/proc_macro.rs

petrochenkov removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 13, 2025

cjgillot self-assigned this Aug 13, 2025

This comment has been minimized.

Sign in to view

This comment was marked as resolved.

Sign in to view

This comment has been minimized.

Sign in to view

rust-bors Bot added a commit that referenced this pull request Aug 13, 2025

Auto merge of #145354 - Kobzol:cache-proc-derive-macros, r=<try>

b712f62

Cache derive proc macro expansion with incremental query

This comment has been minimized.

Sign in to view

This comment was marked as resolved.

Sign in to view

cjgillot reviewed Aug 13, 2025

View reviewed changes

Comment thread compiler/rustc_expand/src/proc_macro.rs Outdated

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jan 15, 2026

petrochenkov reviewed Jan 15, 2026

View reviewed changes

Comment thread compiler/rustc_ast/src/tokenstream.rs Outdated

Comment thread compiler/rustc_incremental/src/persist/dirty_clean.rs Outdated

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 15, 2026

Implement incremental caching for derive macro expansions

8fa2f69

Kobzol force-pushed the cache-proc-derive-macros branch from 05781ca to 8fa2f69 Compare January 16, 2026 06:36

rust-bors Bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jan 16, 2026

This was referenced Jan 16, 2026

Rollup of 6 pull requests #151208

Closed

Rollup of 6 pull requests #151210

Merged

rust-bors Bot merged commit 81cc29a into rust-lang:main Jan 16, 2026
11 checks passed

rustbot added this to the 1.94.0 milestone Jan 16, 2026

Kobzol deleted the cache-proc-derive-macros branch January 16, 2026 21:01

This was referenced Jan 16, 2026

Document -Zcache-proc-macros #151227

Merged

Tracking Issue for -Zcache-proc-macros #151364

Open

github-actions Bot mentioned this pull request Jan 28, 2026

Toolchain upgrade to nightly-2026-02-09 flux-rs/flux#1345

Closed

rust-timer added a commit that referenced this pull request Feb 8, 2026

Unrolled build for #151227

4c51425

Rollup merge of #151227 - Kobzol:document-cache-proc-macros, r=jackh726 Document `-Zcache-proc-macros` Forgot about this in #145354. Created a tracking issue in #151364.

petrochenkov mentioned this pull request Feb 12, 2026

Cache declarative macro expansion on disk (for incremental comp.). Based on #128605 #128747

Draft

perry-blueberry mentioned this pull request Jun 11, 2026

Add -Zcache-proc-macros flag to config_fast_builds.toml bevyengine/bevy#24595

Merged

Uh oh!

Uh oh!

Conversation

Kobzol commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Aug 13, 2025

Uh oh!

Kobzol commented Aug 13, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment was marked as resolved.

This comment has been minimized.

Kobzol commented Aug 13, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment was marked as resolved.

Uh oh!

Kobzol commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Kobzol commented Aug 25, 2025

Uh oh!

Kobzol commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

petrochenkov commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Kobzol commented Aug 25, 2025

Uh oh!

Kobzol commented Jan 15, 2026

Uh oh!

Uh oh!

Uh oh!

petrochenkov commented Jan 15, 2026

Uh oh!

Kobzol commented Jan 16, 2026

Uh oh!

rust-bors Bot commented Jan 16, 2026

Uh oh!

Uh oh!

futile commented Jan 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Kobzol commented Aug 13, 2025 •

edited

Loading

Kobzol commented Aug 15, 2025 •

edited

Loading

Kobzol commented Aug 25, 2025 •

edited

Loading

petrochenkov commented Aug 25, 2025 •

edited

Loading