diff --git a/.gitignore b/.gitignore index a2cfb9d562..2e27323f90 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,10 @@ build/ build_plain/ build_plan/ +# Any out-of-tree build directory, however named. `build/` alone let a second +# configuration (build-assert/) be added by `git add -A`. +build*/ +build-assert/ install/ # TileLang ST standalone build outputs (see temp_docs/standalone_st.md) diff --git a/docs/designs/ptoas-unified-sync-design.md b/docs/designs/ptoas-unified-sync-design.md new file mode 100644 index 0000000000..303b80f3cd --- /dev/null +++ b/docs/designs/ptoas-unified-sync-design.md @@ -0,0 +1,525 @@ +# PTOAS Unified Intra-Core Sync Allocator Design + +## Intent + +`--enable-unified-sync` allocates intra-core synchronization for one function +across all three hardware mechanisms from a single model, instead of committing to +a mechanism before the cost of that choice is known. + +The existing passes each own one mechanism. `--enable-insert-sync` allocates event +ids and falls back to a `PIPE_ALL` barrier when a pool is exhausted. +`--enable-bufid_sync` allocates buffer-id tokens (see `docs/bufid_sync_a5_design.md`). +Neither can trade one resource against the other: when a pipe direction runs out of +event ids, insert-sync must serialize with a barrier even where buffer-id tokens are +free. + +This pass makes the mechanism a decision rather than a premise. Hazards whose +direction would overflow its event pool are routed to buffer-id tokens where the +buffer is eligible; what remains is coloured into the event pools by live interval; +anything that still does not fit spills to a barrier. + +The dependence analysis is **shared** with `pto-insert-sync` rather than +reimplemented. Only the assignment of mechanisms and ids differs. That is a +deliberate constraint, not a convenience: it means both passes see the same program +and the same hazards, which is what makes a differential comparison between them +meaningful rather than a comparison of two front ends. + +Non-goals: + +- Cross-core synchronization. `pto.sync.set` / `pto.sync.wait` carry FFTS or + intra-block flag ids from a different id space, and are untouched. +- Replacing any existing pass. This is one more automatic-synchronization mode, + mutually exclusive with the other four and off by default. Passing more than one is + rejected with a diagnostic rather than resolved by precedence. +- Scheduling. The pass does not move compute ops; it only decides how the orderings + the front end reports are enforced. + +## Mechanisms + +| mechanism | ops | pool | cost | +|---|---|---|---| +| event id | `pto.set_flag` / `pto.wait_flag` | 8 per `(src, dst)` pipe direction, disjoint hardware per direction | directed: orders one direction | +| buffer id | `pto.get_buf` / `pto.rls_buf` | one pool of K, shared across directions; K=0 on A3, K=32 on A5 | a ticket protocol on one buffer; co-consumers of one token serialize | +| barrier | `pto.barrier ` | unlimited | orders every anchor pair spanning it | + +Event pools are per direction and physically disjoint: id 0 in `MTE2->V` is a +different flag from id 0 in `V->MTE3`. The buffer-id pool is not -- it is one global +K, so a buffer-id token consumed anywhere reduces what is available everywhere. + +A hazard always ends on some mechanism. There is no resting state in which a +hazard has been decided but carries no resource; `SyncCodegen` fails the pass if a +set/wait pair reaches emission with no event id. + +## Enabling the pass + +```bash +ptoas input.pto --pto-arch=a5 --enable-unified-sync -o output.cpp +``` + +`--enable-unified-sync` is mutually exclusive with the other four automatic modes -- +`--enable-insert-sync`, `--enable-bufid_sync`, `--enable-inject-barrier-all-sync` and +`--enable-graph-sync-solver` -- and passing two is an error rather than a precedence +question. There is no compiler default: a run that passes none of them inserts no +synchronization at all, which is why every measurement in this document names its +mode explicitly. + +K is derived from `--pto-arch`: 0 for a3, 32 for a5. On a3 the routing step is +therefore inert and the pass allocates events and barriers only. + +Functions that already contain explicit synchronization (`pto.set_flag`, +`pto.wait_flag`, `pto.record_event`, `pto.wait_event`) are left untouched. + +### Flags + +Ten user-facing flags ship with the pass. The first two drive it; the rest are the +oracle described under **Oracle gates**. + +| flag | purpose | +|---|---| +| `--enable-unified-sync` | run the allocator | +| `--enable-unified-sync-debug` | print the model, routing, colouring and emission reports to stderr. Off by default: the reports are verbose enough on a mid-sized kernel to bury anything a caller needs. Gate violations print either way, since they precede a failure and would otherwise leave it unexplained | +| `--dump-sync-extract` | the oracle's structured view of the emitted sync ops | +| `--dump-sync-coverage` | the happens-before edges the emitted sync induces; produces a G1 reference profile | +| `--dump-sync-counts` | kernel-body sync-op counts; produces a G4 reference profile | +| `--check-sync-ids` | gate G3, device-id legality | +| `--check-sync-interference` | gate G2, non-interference | +| `--check-sync-coverage=` | gate G1, differential coverage against a reference profile | +| `--check-sync-self-coverage` | gate G1-self, absolute coverage against the dependence analysis | +| `--check-sync-count-floor=` | gate G4, sync-count floor against a reference profile | + +Three further flags exist for tests and triage. All three are `cl::Hidden`, so they +appear under `--help-hidden` and not under `--help`: + +- `--check-sync-ids-inject-fault` feeds G3 a synthetic illegal id, which is how the + gate is shown to fail on a fault rather than passing everything. +- `--check-sync-interference-inject-fault` appends a synthetic record pair to G2's own + list, in two shapes: a compensation record colliding with an unrelated hazard, and a + pair colliding only on a non-first id. Neither shape can be written as input IR, + because both exist only before codegen. +- `--unified-sync-force-mechanism` stamps hazard group 0 off its type-derived + mechanism onto buffer-id, so the value and its plumbing are exercised on kernels the + router declines. It accepts `bufid` and `bufid-spill`; any other value is an error. + Note that stamping is not routing -- routing is a per-buffer decision this flag does + not make -- so `bufid` deliberately ends in a codegen refusal. + +## Pass placement + +The pass runs as a nested function pass in the same slot as `pto-insert-sync`, +selected by an `else if` chain over the five automatic modes. It therefore sees the +same IR the incumbent sees, which is a precondition for the differential gates. + +The slot itself is constrained: synchronization runs BEFORE +`PTOResolveBufferSelect`, so it still sees per-use `pto.multi_tile_get` operations and +can keep their slot identity for alias and event-id analysis. Moving it later would +erase the distinction rotation depends on. + +## Flow + +The order of the four steps is the design, not an implementation detail. + +1. **Settle every operation before any id exists.** Loop-carried compensation + synthesis runs here: a hazard whose consumer precedes its producer in program + order needs a head-set before the loop and a tail-wait after it, or the first + iteration waits on a flag nobody set. An id assigned to an op that later moves or + disappears is worse than no id, so nothing is allocated until the op set is final. +2. **Route buffers whose direction would overflow.** For each buffer, if its + direction's peak concurrent demand exceeds the event pool and the buffer is + eligible, every hazard on that buffer moves to buffer-id. Routing is + all-or-nothing per hazard: a hazard that took a token on one side and an event id + on the other would be split across two mechanisms, and a split pair hangs. +3. **Colour the rest into the event pools.** Per direction, hazards are coloured by + live interval, so two hazards whose intervals are disjoint share an id. Where the + pool cannot hold the peak, the hazard spills to a `PIPE_ALL` barrier. +4. **Emit, then check.** Emission reuses `SyncCodegen` for events and barriers and + the `bufid_sync` codegen for tokens. The oracle gates then run over the result, + and a gate violation fails the pass. + +## Data model + +One `Hazard` per sync group -- the granularity at which the both-halves invariant +must hold. Each carries: + +- the `(src, dst)` pipe direction; +- a live interval in SyncIR index space, from the producer's index to the + consumer's; +- whether a reverse WAR exists on the same buffer, which is what makes buffer-id + more expensive than an event for that buffer: the token protocol orders the + reverse direction too, so co-consumers serialize; +- the mechanism, once decided. + +Buffer clustering is reused from `BufidSyncAnalysis`: one virtual buffer id per clique +of aliasing tiles, computed on the freshly translated SyncIR before any sync op is +inserted. + +Hazards are joined to those clusters with `MemAlias`, the same relation +`BufidSyncAnalysis` itself uses to decide two tiles are the same memory. Three +plausible keys are all wrong. The root allocation is far coarser than a buffer, so it +conflates tiles that do not alias. A `(scope, base address, size)` tuple is only a +BUCKET inside `BufidSyncAnalysis`, confirmed with `MemAlias` afterwards, and it +degenerates outright at this point in the pipeline: base addresses are not yet +assigned, so the address collapses to zero and distinct clusters share one key. +Pointer identity is exact but incomplete, because the tile dedup drops the losing +`BaseMemInfo` objects. The implementation therefore tries pointer identity first and +falls back to `MemAlias`. + +## Oracle gates + +The allocator is written from scratch, so it cannot be validated by byte-comparing its +output against the incumbent -- and in fact does not reproduce it op for op, though its +counts match on every kernel in the sample corpus (see **Equivalence to the +incumbent**). Correctness is +defined instead by gates over the extracted sync. Two are differential -- they need a +reference profile from a second compiler run -- and four are absolute. + +| gate | kind | asserts | +|---|---|---| +| G1 | differential | every ordering the reference establishes is also established here | +| G1-self | absolute | every dependency `DepBetween` reports is ordered by some emitted sync | +| G2 | absolute | no two overlapping intervals share a sync id; no leaked or unclosed id | +| G3 | absolute | every id is legal for its direction, op and arch | +| G4 | differential | sync-op counts do not regress, per class: total, barriers, `PIPE_ALL` barriers | +| macro reservation | absolute | no compiler hazard takes an id a macro's library implementation uses internally | + +The last has no flag: it runs on every `--enable-unified-sync` compilation and fails +the pass, because no other gate can see the fault. G3 checks that an id is in range, +not that something reserved it, and G2 cannot see an id consumed inside a library +call. + +G1-self is the load-bearing one, because it is the only gate whose green result +means something in its own right. G1 green means "matches the incumbent" and +its strength is bounded by that reference. G1-self builds its expected set by +running the translator and calling `MemoryDependentAnalyzer::DepBetween` directly +over the resulting def/use vectors. Nothing in it reads the incumbent's decisions. +It deliberately does not call `IsMemInfoHasDependency`, which layers sync policy on +top of `DepBetween`: policy is what the gate audits, so policy cannot be part of +the oracle. + +Running a differential is two runs and a file: + +```bash +ptoas k.pto --pto-arch=a5 --pto-level=level3 --enable-insert-sync --dump-sync-coverage 2> ref.cov > /dev/null +ptoas k.pto --pto-arch=a5 --pto-level=level3 --enable-unified-sync --check-sync-coverage=ref.cov +``` + +Both reference dumps go to **stderr**, which matters: redirecting stderr away +discards the profile and leaves a gate that silently checks nothing. + +## Known limitations + +Everything currently known to be incomplete, in one place. Each entry states +whether it is a gap in the allocator or in the gate that checks it, and whether the +incumbent shares it. + +### 1. A larger macro kernel could be pushed into the spill path + +Macro ops are now supported: the (direction, id) pairs a macro's library +implementation consumes internally are reserved over the call's span, padded one +SyncIR index each side, and the colourer treats them as occupancy. + +What remains is a second-order effect. A reservation shrinks effective pool capacity, +so a kernel that fit in eight ids can be pushed into the spill path -- and on A3, +where K is 0, no buffer-id routing can absorb that. No macro kernel in the sample +corpus overflows, spills or routes, because those kernels are small. One with more +concurrently live hazards per direction could be pushed over, which is why the spill +path remains and remains tested. + +### 2. G2 does not validate rotating ids from the command line + +Two gaps, both about rotating (multi-id) hazards. Neither is a live defect, because +the colourer hands out disjoint ids per direction -- but that is a property of the +current colourer, not an invariant. + +- The standalone `--check-sync-interference` gate does not validate rotating event + ids; only the in-pass check does. The CLI gate reads the emitted-IR view, where + `set_flag_dyn` / `wait_flag_dyn` carry a runtime id recorded as `-1`, so those ops + never enter its id check. The ids are statically recoverable -- the emitted selector + is an N-way `arith.select` chain over constants -- so closing this is mechanical. +- Order-dependent conflicts between two hazards within one SyncIR element are + unresolved for rotating ops in either view. The syncops view orders by `irIndex`, + which is element-granular; the emitted-IR view has true program order but cannot + see the rotating ops. For rotating hazards neither view is the authority on order. + +### 3. Same-pipe dependencies are reported uncovered, identically in both modes + +Summed over `test/samples`, across 124 functions producing a gate line, G1-self reports +433 uncovered dependencies: 429 same-pipe, 4 cross-pipe. **The totals are identical +under `--enable-insert-sync`** -- 433, 429 and 4 again -- so this is inherited, not +introduced by this pass. The equality is the load-bearing part; the magnitudes track +the size of the sample corpus and move when it grows. + +They are dominated by A3 `PIPE_V -> PIPE_V` pairs between vector ops: on +`gqa_attention_block` all 61 uncovered dependencies are of that shape, with no +cross-pipe pair among them. Neither pass emits sync for them, and whether they need any +is unresolved. The gate does not distinguish a same-buffer pair from a different-buffer +one, so resolving them means classifying them by allocation identity first. + +### 4. G1-self never enumerates self-pairs + +The pair loop is `for j = i + 1`, so a compound is never paired with itself and a +**same-op loop-carried hazard** -- op x in iteration k against the same op in k+1 -- is +never asked about. Such a pair is absent from every bucket alike: not in +`dependencies`, so not in `covered`, `uncovered`, `unmappable`, `armExcluded` or +`archGuaranteed`. It can neither fail nor pass; it is outside the question. + +Most such pairs are writes to one tile buffer, which program order already orders. The +case that matters is a `tstore` whose iterations write one loop-invariant GM region -- +an `MTE3 -> MTE3` carried WAW this gate does not excuse. On `overflow_writeback` it is +ordered, but transitively and by accident of that kernel's shape, through the carried +`MTE3 -> V` pair composed with the in-iteration `V -> MTE3` pair. A kernel that stores +without that round trip would have the same hazard, no chain, and this gate would still +say nothing. + +The count is not reported by the gate, because a pair outside the question is in no +bucket to be counted; establishing it needs instrumentation the gate does not carry. + +Closing it is a widening, not a bug fix, and it is not free: same-op carried WAW on a +tile buffer is the common case, so enumerating self-pairs needs a same-tile-buffer +exemption the check does not have. Without one, every `tload` writing its tile each +iteration would report uncovered. + +### 5. G1 is one-sided when the reference emits more barriers + +A `PIPE_ALL` barrier at slot `s` contributes `s * (n - s)` forward orderings by +itself, so a reference that discharges hazards with barriers has an edge set denser +by construction than one built from events and tokens. A candidate needing fewer +barriers therefore reports missing orderings for a *reduction* in +over-synchronization, and the superset requirement cannot distinguish that from a +real regression. + +This is not hypothetical, and it is pinned rather than described. On +`overflow_writeback` (a5, level3) G1 reports 665 missing orderings while this run's +edge set is a strict subset of the reference's -- the reverse comparison reports zero +violations, so nothing is established here that the reference lacks. The reference +emits 8 in-body `PIPE_ALL` barriers where this pass emits none, because routing keeps +the event pool from overflowing, and those account for the entire gap. G4 reads the +same fact as an improvement: 55 sync ops against 63. Both directions of the comparison +are pinned in `sync_forced_overflow.pto`. + +The verdict is deliberately left failing, and the gate draws no conclusion from the +barrier counts it prints. The predicate "the reference emits more barriers" also +holds when the candidate is simply broken -- an empty allocation shows it too, since +the reference carries the auto-sync tail barrier and an empty one does not -- so +keying the verdict on it, or even explaining it away at the failure site, would +excuse the exact fault class the gate exists to catch. When the barrier counts +differ, G1-self is the gate to read. + +The violation count is also not a count of orderings lost: `checkCoverageSuperset` +compares raw edge sets with no transitive closure, unlike G1-self, and compares +forward and carried sets separately. Both effects inflate it. Treat it as a +tripwire, not a measurement. + +### 6. The A5 PIPE_V exemption is keyed on the pipe alone + +G1-self treats an A5 `PIPE_V -> PIPE_V` dependency that emitted sync did not cover +as satisfied by the target rather than as a gap. What it rests on is that neither pass +emits sync for such a pair on any A5 kernel measured, so demanding one would make this +gate report every such pair on every A5 kernel it sees. That is an observation about the +kernels available, not a target guarantee this document can cite. + +The test asks only whether both endpoints sit on `PIPE_V`; it does not distinguish a +same-allocation pair from a cross-allocation one. A substantial share of the pairs it +excuses span two distinct allocations that overlap in memory, which is how aliasing +views are expressed once addresses are assigned, so the exemption is broader than a +per-allocation test would be. + +It is deliberately not narrowed: the incumbent treats those cross-allocation pairs +identically, excusing all of them and covering none, so the exemption models the +behaviour of both allocators rather than a divergence between them. Narrowing it would +move a large number of corpus dependencies from excused to uncovered in both modes at +once, changing what the gate reports without changing what either pass emits. + +### 7. Rotation depth greater than one is barely exercised + +A rotating hazard uses `pto.alloc_multi_tile` plus `multi_tile_get`, emitting +`set_flag_dyn` / `wait_flag_dyn` with an N-way `arith.select` chain over constant +ids indexed by `slot % N`. G1-self models these at distance N, recovered from the +selector's own modulus. + +In-tree coverage is thin. Most files using `multi_tile_get` or `alloc_multi_tile` +declare a3, and most do not compile at `--pto-level=level3`, so the rotating path is +predominantly an A3, pre-level3 shape. + +The interaction with buffer-id routing -- which exists only where K > 0, that is on A5 +-- is reached but not exercised. `sync_event_rotation.pto` is an A5 rotating kernel, so +the router does examine its buffer: the rotating hazard reports `revwar=yes`, which +prices a token at parity with an event rather than above it, and routing declines it +only for want of overflow. A rotating kernel that also overflowed would be the first +input to combine the two, and none exists in the tree. + +### 8. Buffer-id rotation is not expressible in this ISA + +Rotation requires selecting a resource id at runtime. Events support it: the dialect +has `SetFlagDynOp` and `WaitFlagDynOp` alongside the immediate forms. Buffer-id does +not, and this is a property of the ISA rather than a missing feature of the pass: +`pto.get_buf` declares `buf_id` as an `I32Attr` -- a compile-time attribute with no +operand form -- there is no `GetBufDynOp`, and the only intrinsics the emitters +produce are `llvm.hivm.GET.BUFI.mode` and `llvm.hivm.RLS.BUFI.mode`, both immediate, +with no register counterpart. + +Buffer-id is therefore a distance-one mechanism here: no per-slot rotation of a token +is expressible. + +What this does NOT establish is that a rotating hazard must never be routed. A token is +per aliasing CLIQUE, not per slot -- a multi-slot buffer's slots fall in one cluster, so +one token would cover the whole rotation rather than needing one per slot. Whether that +is sound is unresolved, and it is not currently decided by anything: the router has no +test on rotation depth, and the case is unreached only because no rotating kernel +overflows. If one ever does, this is the question to settle before trusting the +result. + +### 9. The A3 same-pipe barrier population is not reducible in-house + +A same-pipe hazard cannot use an event, because an event names two distinct pipes. +It is therefore resolved by a barrier or by a target guarantee. On A5 the guarantee +applies to `PIPE_V` (limitation 6). On A3 nothing is dropped, so the barriers are +emitted, and both allocators emit the same ones -- `CanPrunePipeVBarrier` sits in +`InsertSyncAnalysis`, which both pipelines run. + +That prune accepts a `PIPE_V -> PIPE_V` pair only when it did not arrive from the +loop back-edge phase, both endpoints are `PIPE_V`, every dependency pair describes +the exact same access, that access is provably contiguous, and the producer's repeat +is at least `kPipeVPruneMinRepeat` (16). `repeat` is derived from the access shape as +`ceil(validElems / (256 / elementBytes))` -- the number of 256-byte vector-register +passes the producer takes -- not from any loop trip count. + +The population is substantial and dominated by `PIPE_V`. Across the a3 sample kernels +that emit any same-pipe barrier -- 49 of them -- there are 267 in total: 208 `PIPE_V`, +42 `PIPE_M`, 16 `PIPE_MTE3` and one `PIPE_MTE1`. Most `PIPE_V -> PIPE_V` candidates the +predicate examines are refused rather than pruned, and the refusals divide between not +being the exact same access, arriving from the back-edge phase, and a producer repeat +below the threshold. These counts scale with the sample corpus and are cited for +magnitude, not as fixed values. + +What the population costs, as a bound: deleting the same-pipe barriers from emitted +C++ without replacing them takes rmsnorm a3 from 16.10 us to 14.93 us and +rope_kv_cache a3 from 5.57 us to 4.38 us. These figures bound what the barriers cost. +They are not a statement that any particular barrier is unnecessary. + +Three things would let the question be settled. None is obtainable from this +repository: + +- **The content of issue 646.** `test/lit/pto/issue646_pipev_repeat_prune.pto` pins + the threshold's behaviour -- a producer repeat of 15 keeps the barrier, 16 prunes it + -- and also pins keep-cases for a non-contiguous access and for a shared temporary. + The commit that introduced the prune and that test carries a subject line and no + body. The issue is in an external tracker and is referenced nowhere in this tree. +- **A definition of "interval".** The vector op references give per-op A2/A3 figures + of the form "throughput: 2 cycles/repeat (f32), 4 cycles/repeat (f16); interval: 18 + cycles". The term is not defined in those documents nor in the cost-model + reference, so the figure cannot be converted into a required repeat. +- **Hardware, or a simulator that models the hazard.** See limitation 10. + +### 10. The simulator detects cross-pipe sync removal, not same-pipe barrier removal + +An output comparison is only evidence if the comparison is capable of failing, so both +directions were established with control experiments rather than assumed. + +**It detects cross-pipe removal, down to a single pair.** On `post_rmsnorm` a3 the +`PIPE_MTE2 -> PIPE_V` pair that separates a `TLOAD` from the `TMUL` reading that +tile is the only op ordering them. Removing just that pair changes the stored result +(output digest `5b079fd5` to `e657a1cd`); removing all six pairs on that direction +changes it again (`327898c6`). + +**It does not detect same-pipe barrier removal.** A purpose-built kernel with a +same-buffer `PIPE_V -> PIPE_V` RAW chain at producer repeat 15 -- `TADD` writing a +tile, `TROWEXPANDDIV` reading it, with the compiler-inserted `pipe_barrier(PIPE_V)` +as the only separator between them -- produces an identical result with and without +that barrier (`74d57487` either way), on output that is non-degenerate and +input-dependent. `issue646_pipev_repeat_prune` asserts the barrier is required at +repeat 15, so this is a removal the tree treats as unsafe, and it is invisible here. + +This is not merely a serialised pipe model: in a collected trace, five of six +consecutive vector-pipe operations overlap in time, so intra-pipe concurrency is +represented in the timing model. + +**Consequence.** An output comparison can support a claim about cross-pipe +synchronisation. It cannot support one about same-pipe barriers in either direction: +agreement is not evidence of safety, and it cannot be used to argue that a barrier is +removable. A same-pipe claim needs an ordering argument, not a simulator run. + +Two further properties of the harness constrain any such comparison: + +- The generated `golden.py` fills most input tensors with zeros. Comparing outputs + across variants on all-zero input is vacuous, and poisoning an output buffer with + zeros is a no-op when the true output is also zeros. Randomised inputs and a + non-zero poison pattern are both needed for the comparison to mean anything. +- Some kernels cannot serve as an oracle at all. `Qwen3DecodeA3/rmsnorm.pto` at + `--pto-level=level3` does write its output buffer -- a `0xFF` poison is cleared -- + but writes all zeros whatever the input. + +## Testing + +Eight lit files drive `--enable-unified-sync`, across 34 RUN lines that invoke it. + +| file | pins | +|---|---| +| `sync_unified_model.pto` | the data model, the chosen ids, and that hazards on different buffers report DIFFERENT clusters -- the check that separates the `MemAlias` join from a degenerate key | +| `sync_mechanism_field.pto` | that mechanism is derived from type rather than defaulted flat, that it is stamped on both halves of a hazard, and that the spill drops type and mechanism together | +| `sync_extract_oracle_views.pto` | both extractor renderings, their field names and their program order; the only test that would notice a renamed field | +| `sync_forced_overflow.pto` | forced event-pool overflow as a dial, the routing predicate discriminating written-back from forward-only, and both directions of G1's barrier-delta comparison | +| `sync_overflow_id_gap.pto` | buffer-id routing ABSORBING an overflow, so nothing spills, and that no hazard reaches codegen without a realised mechanism | +| `sync_unified_macro_pinning.pto` | that a macro's library-internal event ids are reserved before colouring, that the emitted id matches the incumbent's, and separately that the reservation was actually seeded | +| `sync_unified_compensation_g2.pto` | that the G2 compensation exemption scopes to a hazard's own compensation pair and no further | +| `sync_event_rotation.pto` | rotating pairs at depth 2, that both ids are primed ahead of the loop, and that G1-self models them at distance N | + +Each of the eight has been shown to fail when the behaviour it pins is deliberately +broken, so none of them passes vacuously. + +The gate fixtures are separate and do not drive the allocator: +`sync_oracle_self_validation.pto` and the `sync_gate_g*.pto` family pin that each gate +fails on an injected fault. + +No kernel in the sample corpus overflows an event pool -- per-direction peak demand +stays inside the pool of 8 on every one measured -- so the overflow, routing and spill paths would be unreachable +without synthetic inputs that overflow on purpose. That is what `sync_forced_overflow.pto` +exists for, and why it pins a non-overflowing control alongside the overflowing case: a +synthetic that always overflows could be broken rather than saturated. + +## Equivalence to the incumbent + +Three claims of different strength, kept apart deliberately. Op-count identity does not +entail cycle identity, and the cycle measurements below show a case where it does not. + +### Proven: sync-op count parity across the corpus + +The allocator replaces only the assignment of mechanisms and ids; the dependence +analysis, hoisting and redundancy pruning ahead of it are shared. Comparing emitted PTO +IR between `--enable-insert-sync` and `--enable-unified-sync`, at the highest level each +kernel compiles at, over the sample corpus: of 130 kernels, 115 emit under both modes -- +56 byte-identical, 59 differing, 15 emitting under neither. No kernel fails under the +unified allocator that succeeds under the incumbent. + +Every one of the 59 differences is at an IDENTICAL per-kernel sync-op count: 2242 ops on +each side across those 59, and 2603 across all 115 that emit -- a delta of zero either +way. The differences are ordering, not quantity. This holds for A5 as well as A3: all 21 +A5 kernels compile at level3 under both modes, 20 differing and 1 identical, every one at +an equal count. + +### Measured: two A5 kernels, in the cycle domain + +Ordering is not free, so op-count parity is not a cycle result. Two A5 kernels were +measured on the cycle-accurate model, per-core latency, with a determinism control: + +| kernel | insert-sync | unified-sync | delta | +|---|---|---|---| +| `rmsnorm` (vector) | 18.35 us | 18.38 us | +0.16% | +| `qwen3_decode_incore_10` (cube) | 82.15 us | 82.15 us | 0.00% | + +The control matters: repeating the `rmsnorm` insert-sync run reproduced 18.35 us with a +bit-identical instruction-cycle sum, so +0.16% is a signal rather than run variance. Only +the SoC tick counter moves between repeats, and it is not used here. + +Note that the two available metrics disagree in direction on `rmsnorm`: unified's summed +instruction cycles are LOWER by 3.2% while its per-core latency is marginally higher. The +trace span is the metric to trust -- a schedule can retire fewer cycles in total and still +finish later -- and quoting the cycle sum instead would turn a small regression into an +apparent win. + +### Not claimed: cycle neutrality as a general property + +Two kernels are not the corpus. What the measurements support is that reordering at equal +op count costs little on the kernels tested -- nothing worse than a fraction of a percent, +in one case a regression and in the other nothing. They do not establish that the +allocator is cycle-neutral in general, and no claim here depends on it being so: the +allocator's purpose is to make the mechanism a decision rather than a premise, and on the +production corpus it declines to route because no kernel exhausts a pool. diff --git a/include/PTO/Transforms/InsertSync/SyncCodegen.h b/include/PTO/Transforms/InsertSync/SyncCodegen.h index 9502a60123..9b9324736f 100644 --- a/include/PTO/Transforms/InsertSync/SyncCodegen.h +++ b/include/PTO/Transforms/InsertSync/SyncCodegen.h @@ -38,8 +38,15 @@ class SyncCodegen { /// 入口函数：执行代码生成 void Run(); + + /// True if a set/wait hazard reached emission with NO event id, i.e. with no + /// mechanism realised. `Run()` emits a diagnostic for each such hazard; the caller + /// decides whether that fails the pass. See the guard in `SyncInsert`. + bool sawUnrealisedHazard() const { return sawUnrealisedHazard_; } private: + bool sawUnrealisedHazard_ = false; + // --- 核心插入逻辑 --- void SyncInsert(IRRewriter &rewriter, Operation *op, SyncOperation *sync, bool beforeInsert); diff --git a/include/PTO/Transforms/InsertSync/SyncCommon.h b/include/PTO/Transforms/InsertSync/SyncCommon.h index 9a99103b73..ccfbb7e4c8 100644 --- a/include/PTO/Transforms/InsertSync/SyncCommon.h +++ b/include/PTO/Transforms/InsertSync/SyncCommon.h @@ -163,8 +163,48 @@ class SyncOperation { SYNC_BLOCK_WAIT, SYNC_BLOCK_ALL, }; - + + /// Which hardware resource backs this sync -- and therefore, per the buf-id + /// design revision, WHERE it may be placed. + /// + /// This is deliberately a DIFFERENT axis from TYPE. `TYPE` is the op *shape* + /// (a set, a wait, a barrier) and is decided by the analysis. `MECHANISM` is + /// the resource *class* that backs it, and is decided by the allocator: + /// + /// EVENT Per-(src,dst) private pool of 8 flags. Carried at a SyncIR + /// position, so `MoveSyncState` may hoist it out of a branch or + /// loop. This is what every in-tree sync is today. + /// BUFID Shared rotating pool of K buffer-ID tokens (K=0 on A3, 32 on A5). + /// A token's get/rel counters are BIDIRECTIONAL: one token discharges + /// the forward RAW *and* the reverse WAR. Two consequences, and they + /// are the reason this enum exists rather than a bool: + /// - no backward-matched pair is needed (`UpdateBackwardMatchSync` + /// lives in SyncEventIdAllocation, which the unified allocator + /// never runs -- so there is nothing to delete); + /// - it MUST be emitted OP-ANCHORED, bracketing the access, not at + /// a hoisted SyncIR position. + /// Set by the allocator when it routes a hazard off events. + /// BARRIER Spill. Carries no id. + /// BLOCK Cross-core block sync (reserved ids 14/15). Not routed by this + /// allocator; present so a block sync can never be mistaken for an + /// event. + /// + /// Every value EXCEPT BUFID is derivable from TYPE, and the constructor does + /// derive it (see `DefaultMechanismFor`), so no construction site has to + /// remember to set it and a barrier can never silently claim to be an event. + /// BUFID is the one value that is *not* derivable -- buffer-ID sync does not + /// go through `SyncOperation` at all today -- and it is precisely the routing + /// decision the unified allocator exists to make. + enum class MECHANISM { EVENT, BUFID, BARRIER, BLOCK }; + bool isCompensation = false; + /// For a compensation op (head-set / tail-wait), the `kSyncIndex_` of the in-body + /// hazard it primes and drains; -1 otherwise. Needed because a compensation pair is + /// created as its OWN sync group, so `GetSyncIndex()` does not identify its owner -- + /// and G2 must be able to tell "these two records are halves of one hazard" from + /// "two different hazards collide on an id". Without it the exemption would have to + /// excuse every compensation record, which would drop real conflicts. + int compensationOf = -1; static const int kNullEventId{-1}; @@ -173,7 +213,20 @@ class SyncOperation { // Root buffers participating in the dependency that created this sync pair. // These are kept for allocation/widening heuristics and debug printing; set/ // wait redundancy pruning is based on the pipe pair semantics. + // + // `rootBuffer` is the ROOT ALLOCATION, which is much COARSER than a buffer: many + // distinct tiles share one root. It is therefore NOT a buffer identity, and must + // not be used as one -- see `depMemInfos` below. SmallVector depRootBuffers; + // The BaseMemInfo objects behind this dependency, kept so a consumer can + // recover the TILE identity `(scope, baseAddr, size)` rather than the coarse + // root above. Owned by the translator's `Buffer2MemInfoMap`; these are raw + // pointers into that arena, which every analysis in the pipeline shares. + // + // This exists because `InsertSyncAnalysis` HAS these pointers when it builds a + // sync pair and used to discard them, keeping only the roots -- which left the + // unified allocator unable to tell which buffer a hazard was actually on. + SmallVector depMemInfos; bool uselessSync{false}; int eventIdNum{1}; // For multi-buffer dyn-event sync: the slot SSA expression at this access @@ -192,15 +245,25 @@ class SyncOperation { SyncOperation(TYPE type, pto::PipelineType srcPipe, pto::PipelineType dstPipe, unsigned kSyncIndex, unsigned syncIRIndex, std::optional forEndIndex, bool isComp = false) - : isCompensation(isComp), eventIds({}), type_(type), srcPipe_(srcPipe), + : isCompensation(isComp), eventIds({}), type_(type), + mechanism_(DefaultMechanismFor(type)), srcPipe_(srcPipe), dstPipe_(dstPipe), kSyncIndex_(kSyncIndex), syncIRIndex_(syncIRIndex), forEndIndex_(forEndIndex) {}; - + ~SyncOperation() = default; - + std::unique_ptr GetMatchSync(unsigned index) const; - + TYPE GetType() const { return type_; } + + /// The resource class backing this sync. Defaults from TYPE; only the + /// allocator moves a sync off its default (today: onto BUFID). + MECHANISM GetMechanism() const { return mechanism_; } + void SetMechanism(MECHANISM m) { mechanism_ = m; } + + /// The mechanism a given TYPE implies. Kept next to the enum so the two can + /// never drift; `SetPipeAll` and the constructor are its only callers. + static MECHANISM DefaultMechanismFor(TYPE t); pto::PipelineType GetSrcPipe() const { return srcPipe_; } pto::PipelineType GetDstPipe() const { return dstPipe_; } @@ -223,6 +286,7 @@ class SyncOperation { bool isBarrierType() const; static std::string TypeName(TYPE t); + static std::string MechanismName(MECHANISM m); std::string GetCoreTypeName(TCoreType t) const; using SyncOperations = @@ -235,6 +299,9 @@ class SyncOperation { private: TYPE type_; + // Private on purpose: it must stay consistent with type_ (see SetPipeAll, + // which downgrades both together). Write it through SetMechanism only. + MECHANISM mechanism_; pto::PipelineType srcPipe_; pto::PipelineType dstPipe_; const unsigned kSyncIndex_; diff --git a/include/PTO/Transforms/InsertSync/SyncEventIdAllocation.h b/include/PTO/Transforms/InsertSync/SyncEventIdAllocation.h index d576061cde..44a15ae798 100644 --- a/include/PTO/Transforms/InsertSync/SyncEventIdAllocation.h +++ b/include/PTO/Transforms/InsertSync/SyncEventIdAllocation.h @@ -40,6 +40,12 @@ using SyncCycle = DenseMap; class SyncEventIdAllocation { public: + /// Number of event ids reserved at the tail of the intra-core pool + /// [0, kTotalEventIdNum) for the direction `src -> dst`; 0 when the direction + /// reserves nothing. Exposed so the device-id-legality gate reads the same + /// reservation map this allocator obeys, rather than duplicating it. + static uint64_t GetReservedEventIdNum(PipelineType src, PipelineType dst); + SyncEventIdAllocation(SyncIRs &syncIR, SyncOperations &syncOperations) : syncIR_(syncIR), syncOperations_(syncOperations) { reserveBlockAllEventIds(); @@ -124,6 +130,7 @@ class SyncEventIdAllocation { static const llvm::DenseMap, uint64_t> reservedEventIdNum; uint64_t reservedBlockSyncEventIdNum{0}; + }; } // namespace pto diff --git a/include/PTO/Transforms/InsertSync/SyncOracleExtract.h b/include/PTO/Transforms/InsertSync/SyncOracleExtract.h new file mode 100644 index 0000000000..b3224953b3 --- /dev/null +++ b/include/PTO/Transforms/InsertSync/SyncOracleExtract.h @@ -0,0 +1,136 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===- SyncOracleExtract.h - Shared sync-op extractor (oracle) ---*- C++ -*-===// +// +// The shared walk-surface the oracle gates ride on: the emitted `pto.set_flag` / +// `pto.wait_flag` / `pto.get_buf` / `pto.rls_buf` / `pto.barrier` ops of one +// function. G3 reads the ids, G2 reads their live ranges. +// +// `SyncOpRecord` describes the same synchronization before codegen, as an allocator +// holds it. Only G3 has an overload taking it, and only the gate self-test feeds it, +// because nothing else in this build produces pre-codegen records. +// +// Extraction is deterministic (program order), so a rendering is stable across runs. +//===----------------------------------------------------------------------===// + +#ifndef MLIR_DIALECT_PTO_TRANSFORMS_INSERTSYNC_SYNCORACLEEXTRACT_H +#define MLIR_DIALECT_PTO_TRANSFORMS_INSERTSYNC_SYNCORACLEEXTRACT_H + +#include "PTO/Transforms/InsertSync/SyncCommon.h" +#include "mlir/Dialect/Func/IR/FuncOps.h" +#include "llvm/ADT/ArrayRef.h" +#include "llvm/ADT/STLFunctionalExtras.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/ADT/StringRef.h" +#include "llvm/Support/raw_ostream.h" + +#include + +namespace mlir { +namespace pto { +namespace oracle { + +/// One emitted synchronization op, read out of the IR after codegen. +/// `eventId` is set only for set_flag/wait_flag; `bufId` only for +/// get_buf/rls_buf; both are -1 when not applicable. +struct IRSyncRecord { + std::string opName; + std::string srcPipe; + std::string dstPipe; + /// Typed pipes for the gates. The strings above are for rendering only; + /// gates must never string-match a pipe name. + PipelineType srcPipeType = PipelineType::PIPE_UNASSIGNED; + PipelineType dstPipeType = PipelineType::PIPE_UNASSIGNED; + int64_t eventId = -1; + int64_t bufId = -1; + /// get_buf/rls_buf `mode`. 0 = the bidirectional token (the only mode PTOAS + /// emits today). Non-zero = the directional mode (set_flagV2/wait_flagV2), + /// deferred -- see the buffer-ID legality notes in SyncOracleGates.h, which + /// applies the strict pairing rules only to mode 0. + int64_t mode = 0; + /// Enclosing block. A get_buf and its rls_buf must sit in the SAME block, or + /// the pair may not both execute (counter desync). Null for non-buf ops. + mlir::Block *block = nullptr; + /// The op itself, for gates that must ask where it sits in the control flow. An + /// event pair is checked on its enclosing conditionals, which a block pointer + /// alone cannot answer: two blocks differ for loop nesting too, and that case is + /// legitimate. + mlir::Operation *op = nullptr; + unsigned order = 0; // program order within the function +}; + +/// One entry of the in-memory SyncOperation set, before codegen. `eventIds` is +/// empty until an allocator assigns them. +struct SyncOpRecord { + std::string type; + /// `static_cast(SyncOperation::TYPE)`; -1 when unset. Gates switch on + /// this, never on the `type` string. + int typeCode = -1; + /// `SyncOperation::MechanismName` of the resource class the allocator routed this + /// sync to -- and so whether it is SyncIR-positioned (event/barrier) or op-anchored + /// (bufid). Empty until the pre-codegen extractor fills it. + /// + /// RENDERING ONLY. Unlike `type`, this has no numeric companion, so nothing can + /// switch on it without string-matching a name. A gate that needs the mechanism as + /// a decision should get a `mechanismCode` beside this, the way `typeCode` sits + /// beside `type`, rather than compare these strings. + std::string mechanism; + int srcPipe = -1; + int dstPipe = -1; + llvm::SmallVector eventIds; + unsigned syncIndex = 0; + unsigned irIndex = 0; + /// Mirrors `SyncOperation::isCompensation` / `compensationOf`. G2's syncops scan + /// needs both: a compensation record shares its ids with the in-body pair it primes + /// BY DESIGN, so that pairing must not read as two hazards colliding -- while a + /// collision with any OTHER hazard still must. + bool isCompensation = false; + int compensationOf = -1; +}; + +/// Human-readable name of an InsertSync PipelineType ("MTE2", "V", "ALL", ...). +llvm::StringRef pipelineTypeName(PipelineType pipe); + +/// The pipe a `get_buf`/`rls_buf` `op_type` names. +/// +/// `PTO_PipeLikeAttr` admits three spellings -- pipe_event_type, sync_op_type, and a +/// plain PipeAttr. A decoder handling only the first two returns PIPE_UNASSIGNED for +/// the third rather than failing, so its caller silently loses the op. +pto::PIPE bufSyncPipe(mlir::Attribute opTypeAttr); + +/// Render a report and write it to stderr atomically. +/// +/// STDERR, NEVER STDOUT: stdout carries the emitted kernel source, so a report +/// written there is spliced into the C++ and it stops compiling. +/// +/// MLIR runs nested func::FuncOp passes in PARALLEL and ptoas does not disable +/// threading, so unsynchronized writes from several functions interleave mid-line. +/// Every oracle pass must print through here. Ordering *between* functions stays +/// nondeterministic, so no consumer may depend on it. +void emitReport(llvm::function_ref body); + +/// Extract the emitted sync ops from `func`, in program order. +llvm::SmallVector extractFromIR(func::FuncOp func); + +/// The same synchronization before codegen, as an allocator holds it: the ids it has +/// just written, before anything is emitted. +llvm::SmallVector extractFromSyncOps(const SyncOperations &ops); + + +/// Deterministic, diff-friendly rendering (one record per line). +void printIRRecords(llvm::raw_ostream &os, llvm::StringRef funcName, + llvm::ArrayRef records); +void printSyncOpRecords(llvm::raw_ostream &os, llvm::StringRef funcName, + llvm::ArrayRef records); + +} // namespace oracle +} // namespace pto +} // namespace mlir + +#endif // MLIR_DIALECT_PTO_TRANSFORMS_INSERTSYNC_SYNCORACLEEXTRACT_H diff --git a/include/PTO/Transforms/InsertSync/SyncOracleGates.h b/include/PTO/Transforms/InsertSync/SyncOracleGates.h new file mode 100644 index 0000000000..46f13b1ebb --- /dev/null +++ b/include/PTO/Transforms/InsertSync/SyncOracleGates.h @@ -0,0 +1,697 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===- SyncOracleGates.h - Oracle correctness gates -------------*- C++ -*-===// +// +// Correctness gates over the sync a compilation emitted, read through +// SyncOracleExtract.h. Four judge one compilation on its own; two are DIFFERENTIAL +// and compare two runs of the same function, inheriting the reference run's +// completeness as an assumption: +// G3 device-id legality: every id is one the hardware and the compiler's +// own reservations permit, for that op, direction and arch. +// G2 non-interference: no two overlapping intervals share an id. +// G1-self every dependency `DepBetween` reports is ordered by emitted sync. +// G1 coverage superset (differential): every ordering a reference run +// establishes must also be established by this one. +// G4 sync-count floor (differential). +// +// G3's rules, and why each is not already covered elsewhere: +// +// R1 event id range. Intra-core set/wait ids live in [0, kTotalEventIdNum). +// The IR spelling cannot express a violation (`pto.set_flag`'s event_id is +// an EVENT_ID0..EVENT_ID7 *enum*), but the in-memory `SyncOperation:: +// eventIds` is a plain SmallVector that an allocator writes freely. +// That is the surface this rule guards. +// +// R2 per-direction reservation. `SyncEventIdAllocation::reservedEventIdNum` +// reserves the tail of the pool for some directions (today: V<->S reserve +// one id, so only [0, 7) is available there). Representable in real IR, +// so this gate checks it. +// +// R3 block-sync-all reserved ids. When a SYNC_BLOCK_ALL exists, ids +// kBlockSyncAllCubeEventId (14) / kBlockSyncAllVectorEventId (15) belong +// to it alone and the block set/wait pool shrinks to +// [0, kBlockSyncSetWaitEventIdNum - kReservedBlockSyncEventIdNum). +// NOTE: nothing in this build emits a block-sync op, so this rule is +// unexercised. It is encoded because its id space is where a stomp appears. +// +// R4 buffer id range. Buffer ids must lie in [0, 31]. Checked on the emitted +// ops, a second surface after the op verifier: the pre-codegen record type +// cannot represent a buffer id at all. +// +// R5 buffer id arch capacity. Buffer-id sync is exposed only where K > 0 +// (K=0 on A3, K=32 on A5), so a buf-id op with buf_id >= K for the arch +// being compiled is rejected. K reaches the gate from the caller. +// +// Barriers carry no id; an id on a barrier is an allocator bug, so it is +// reported too. +// +// Ops whose ids live in a *different* id space are deliberately out of scope: +// `pto.sync.set` / `pto.sync.wait` carry a cross-core FFTS/intra-block flag id +// (lowered through `createFFTSMsg` / `set_intra_block`), not an intra-core event +// id -- the in-tree corpus legitimately uses values such as 17 there. +//===----------------------------------------------------------------------===// + +#ifndef MLIR_DIALECT_PTO_TRANSFORMS_INSERTSYNC_SYNCORACLEGATES_H +#define MLIR_DIALECT_PTO_TRANSFORMS_INSERTSYNC_SYNCORACLEGATES_H + +#include "PTO/Transforms/InsertSync/SyncCommon.h" +#include "PTO/Transforms/InsertSync/SyncEventIdAllocation.h" +#include "PTO/Transforms/InsertSync/SyncOracleExtract.h" +#include "llvm/ADT/StringMap.h" + +namespace mlir { +namespace pto { +namespace oracle { + +/// Everything G3 needs to know about the target of one compilation. +struct DeviceIdLimits { + /// Size of a per-(src,dst) intra-core event pool. + unsigned intraCoreEventIdNum = kTotalEventIdNum; + /// Size of the block-sync set/wait pool, before block-all reservation. + unsigned blockSyncEventIdNum = kBlockSyncSetWaitEventIdNum; + /// K: buffer ids this arch exposes. 0 on A3, 32 on A5. + unsigned bufIdCapacity = 0; + /// Width of the hardware buf_id field; mirrors `verifyBufSyncOp`'s [0, 31]. + unsigned bufIdHwMax = 32; +}; + +enum class IdViolationKind { + EventIdOutOfRange, + EventIdReservedForDirection, + EventIdReservedForBlockAll, + BlockAllEventIdWrong, + BarrierCarriesEventId, + BufIdOutOfRange, + BufIdExceedsArchCapacity, + EventIdCollidesWithMacroHidden, + /// A (direction, event id) has a different number of `set_flag` and `wait_flag` + /// ops in the emitted IR. Checked on the EMITTED ops because a pre-codegen count + /// includes syncs the emitter may still drop: `MergeSyncList` discards a sync that + /// `IsSyncExist` already finds in the destination list, so two indistinguishable + /// ops merged onto one element leave a wait with no set -- a hang that counting + /// before codegen cannot see. + EventSetWaitUnbalanced, +}; + +llvm::StringRef idViolationKindName(IdViolationKind kind); + +struct IdViolation { + IdViolationKind kind; + unsigned order; // program order (IR view) / sync index (SyncOperation view) + std::string opName; + int64_t id; + std::string detail; // human-readable statement of the rule that was broken +}; + +/// G3 over the emitted IR. +llvm::SmallVector +checkDeviceIdLegality(llvm::ArrayRef records, + const DeviceIdLimits &limits); + +/// G3 over the pre-codegen SyncOperation set -- the ids an allocator just wrote. +llvm::SmallVector +checkDeviceIdLegality(llvm::ArrayRef records, + const DeviceIdLimits &limits); + +//===----------------------------------------------------------------------===// +// G3-macro: collision with a macro's library-internal event ids +//===----------------------------------------------------------------------===// +// +// A macro op lowers to a pto-isa library call that consumes event ids INSIDE the +// call, where no PTO IR names them. `SyncMacroModel::hiddenEvents` records which +// (srcPipe, dstPipe, id) the library uses, so the ids are known even though the +// uses are not representable. An allocator that hands one of those ids to a +// compiler hazard whose interval spans the call produces a kernel where the +// library's wait is satisfied by the wrong producer -- silently, with no op in the +// IR to point at. +// +// THE RESERVATION MIRRORS THE INCUMBENT'S, deliberately, so that "legal" means the +// same thing on both paths: the span runs from the macro's first phase to its last, +// padded one SyncIR index each side, and the pad is what stops a hazard that ends +// exactly where the call begins from being read as disjoint. +// +// WHY THIS GATE EXISTS BEFORE THE FIX. While macro ops are refused there is no way +// to check a reservation by running it -- the kernel never reaches allocation. So +// the detector is built first and shown to fire on the CURRENT wrong id; only then +// is the reservation worth writing, because its green result then means something. +llvm::SmallVector +checkMacroHiddenEventCollisions(const SyncIRs &syncIR, + llvm::ArrayRef records); + +/// Deterministic rendering. `view` is "ir" or "syncops". +void printIdViolations(llvm::raw_ostream &os, llvm::StringRef funcName, + llvm::StringRef view, size_t recordCount, + llvm::ArrayRef violations); + +//===----------------------------------------------------------------------===// +// G4: sync-count floor +//===----------------------------------------------------------------------===// +// +// Differential gate: one run must not synchronize *more* than a reference run of +// the same kernel. Two compilations, two profiles, one comparison. What the +// reference is compiled with is the caller's choice. +// +// The counts below are **kernel-body PTO sync ops**, taken from the IR view -- +// not a grep of the emitted C++. That distinction is load-bearing: ptoas emits a +// `ptoas_auto_sync_tail` helper *definition* into every output regardless of sync +// mode, and its body contains a set_flag/wait_flag pair and a pipe_barrier, so a +// whole-file grep over-counts by a constant. `extractFromIR` walks the function's +// ops, so the boilerplate is structurally invisible to it. +// +// A scalar `total <= reference.total` floor is NOT sufficient. A fully serializing +// allocation can tie on total -- one PIPE_ALL barrier per hazard replaces a set/wait +// pair one for one -- so a scalar floor waves the degenerate through. A barrier can +// even *lower* the op count while strictly worsening the schedule. Hence the floor is +// per class: +// +// F1 total op count must not increase. +// F2 barrier count must not increase (a barrier orders every pipe; a directed +// set/wait pair orders one direction). +// F3 PIPE_ALL barrier count must not increase (full serialization). +// +// A candidate function with no reference profile is a violation, not a pass: +// a gate that silently skips what it cannot find is not a gate. + +struct SyncCountProfile { + unsigned setFlag = 0; + unsigned waitFlag = 0; + unsigned getBuf = 0; + unsigned rlsBuf = 0; + unsigned barrier = 0; // all pto.barrier, any pipe + unsigned barrierAll = 0; // subset of `barrier` whose pipe is PIPE_ALL + unsigned total() const { + return setFlag + waitFlag + getBuf + rlsBuf + barrier; + } +}; + +enum class CountViolationKind { + TotalIncreased, + BarrierIncreased, + BarrierAllIncreased, + MissingReference, +}; + +llvm::StringRef countViolationKindName(CountViolationKind kind); + +struct CountViolation { + CountViolationKind kind; + unsigned candidate; + unsigned reference; + std::string detail; +}; + +/// Kernel-body sync-op counts for one function. +SyncCountProfile computeSyncCounts(llvm::ArrayRef records); + +/// G4. `reference` is InsertSync's profile for the same function. +llvm::SmallVector +checkSyncCountFloor(const SyncCountProfile &candidate, + const SyncCountProfile &reference); + +/// One machine-readable line per function, re-readable by parseSyncCounts(). +void printSyncCounts(llvm::raw_ostream &os, llvm::StringRef funcName, + const SyncCountProfile &profile); + +/// Parse `[sync-count]` lines out of a previous run's output. Other lines are +/// ignored, so a raw stdout capture can be handed straight back in. +llvm::StringMap parseSyncCounts(llvm::StringRef text); + +void printCountViolations(llvm::raw_ostream &os, llvm::StringRef funcName, + const SyncCountProfile &candidate, + const SyncCountProfile &reference, + llvm::ArrayRef violations); + +//===----------------------------------------------------------------------===// +// G2: non-interference +//===----------------------------------------------------------------------===// +// +// Coverage alone cannot see this class of bug. An allocation can order every +// hazard it is supposed to order (G1 green) and still be wrong, because two +// hazards whose lifetimes overlap were handed the SAME id: the first hazard's +// wait can be satisfied by the second hazard's set, and the ordering it was +// meant to establish silently evaporates, and the op count is unchanged, so +// nothing that counts ops can see it either. +// +// The invariant, which generalizes `BufidSyncIdAlloc::validateNoSamePhysicalIdNesting` +// from buffer ids to every id class: +// +// For a given id key, the intervals that use that id must be pairwise +// disjoint -- never nested, never overlapping. +// +// The key differs per id class, and getting it wrong makes the gate either +// blind or wrong: +// +// * event ids: key = (srcPipe, dstPipe, eventId). Event flags are per- +// (src,dst)-direction DISJOINT hardware, so EVENT_ID0 may be live in +// MTE2->V and in V->MTE3 simultaneously. Keying on the id alone would raise +// a false violation on correct code. +// * buffer ids: key = bufId alone. The buffer-id pool is global across pipes, +// matching validateNoSamePhysicalIdNesting, which keys on physicalId only. +// +// Implementation is a linear scan in program order keeping a stack of open intervals +// per key: a set/get pushes and reports any interval already open on that key, a +// wait/rls pops, a pop on an empty stack is a close without an open, and anything +// left on the stack at the end is a leaked id. +// +// A loop-carried pair -- a `wait` satisfied by the PREVIOUS iteration's `set`, so +// the wait precedes the set in program order -- surfaces as `wait-without-set` +// plus `unclosed`. The sync pass makes such a hazard work by priming the flag +// before the loop and draining it after, so on well-formed output neither kind +// fires. An unprimed loop-carried pair blocks forever on iteration 1, and these +// two kinds are the only thing that detects it. Do not weaken them: if cyclic +// matching is ever added it must DISTINGUISH a primed backward pair from an +// unprimed one, not accept both. + +// --- Buffer-ID legality --------------------------------------------------- +// +// These are HARDWARE rules, not design choices: violating them desynchronises +// the id's global get/rel counters, and a get_buf that can never pop HANGS the +// core. They are therefore checked as part of G2, and they are immune to any +// allocator-design decision -- which is why this gate can be built before the +// allocator exists. +// +// B1 no re-acquire of a live id. Covers BOTH the same-pipe case +// same-id nesting: get,get,rel,rel double-increments the counter so a get +// may never pop) AND the cross-pipe same-id interleave ( +// get(MTE2,#0) get(V,#0) rel(MTE2,#0) rel(V,#0) is illegal). Both surface +// as "this id was opened twice before closing". +// B2 no rls_buf without a live get_buf. +// B3 no get_buf left unreleased. +// B4 pipe-matched pairing -- a rls_buf closes a get_buf on the SAME pipe. The +// token brackets one op on one pipe; a cross-pipe close is not a pair. +// B5 an id must span exactly 2 distinct pipes: buffer-ID is +// ONLY for sync between two different pipes; same-pipe ordering needs +// bar.pipe. A single-pipe id synchronises nothing and burns an id. +// B6 an id SHOULD NOT span more than 2 pipes ("not +// recommended" -- soft). WARNING, not an error, and deliberately so: the +// shipping BufidSync pass already emits one id across MTE2/V/MTE3 on real +// kernels. Making it an error would fail on known-good output. +// It is reported because it is the coalescing bound any allocator routing +// hazards onto buffer tokens has to respect. +// B7 a get_buf and its rls_buf must sit in the SAME block. A pair split +// across if/else arms would be matched by any flat program-order walk, yet +// at run time only one arm executes -- the counters desync (the ASC +// alloc/free-balance invariant). Shipping BufidSync anchors both ops on the +// same op, so this holds by construction there. +// +// MODE. get_buf/rls_buf carry a `mode` operand. Mode 0 is the bidirectional +// token -- the only mode PTOAS emits -- and the strict rules above apply to it. +// Non-zero mode is the directional variant (set_flagV2/wait_flagV2), which the +// spec explicitly frees from strict pairing; it is a deferred mechanism, so the +// pairing rules are NOT applied to it and its presence is reported instead. The +// gate keys on `mode` from day one so that mechanism is additive, not a rewrite. + +enum class InterferenceKind { + EventIdNested, + EventIdWaitWithoutSet, + EventIdUnclosed, + /// A set_flag and the wait_flag closing it sit under DIFFERENT conditionals, so + /// some path runs one without the other: a wait with no set hangs, a set with no + /// wait leaks the id. Loop nesting is deliberately not compared -- a set outside a + /// loop priming a wait inside it is the intended idiom, and the zero-trip case is + /// checked on the loop spine instead. + EventIdPairGuardMismatch, + BufIdNested, // B1 + BufIdReleaseWithoutAcquire, // B2 + BufIdUnclosed, // B3 + BufIdPipeMismatch, // B4 + BufIdSinglePipe, // B5 + BufIdTooManyPipes, // B6 (warning) + BufIdPairSplitAcrossBlocks, // B7 + BufIdDirectionalModeUnsupported, // mode != 0 (deferred mechanism) +}; + +/// A warning is reported but does not fail the gate. Used only where the spec +/// itself says "not recommended" rather than "illegal" -- see B6. +enum class Severity { Error, Warning }; + +llvm::StringRef interferenceKindName(InterferenceKind kind); +llvm::StringRef severityName(Severity severity); + +struct InterferenceViolation { + InterferenceKind kind; + Severity severity = Severity::Error; + unsigned order; // program order of the offending op + unsigned openedAt; // program order of the still-live interval, when relevant + std::string opName; + std::string key; // rendered id key, e.g. "MTE2->V id=0" or "buf_id=0" + std::string detail; +}; + +/// G2 over the emitted IR. +llvm::SmallVector +checkNonInterference(llvm::ArrayRef records); + +/// G2 over the pre-codegen SyncOperation set -- the ids an allocator just wrote, +/// before codegen can drop or merge any of them. +llvm::SmallVector +checkNonInterference(llvm::ArrayRef records); + +/// KNOWN LIMITATION: G2 does not check ROTATING (multi-id) hazards at all. +/// `set_flag_dyn` / `wait_flag_dyn` carry a runtime id, recorded as -1, so those ops +/// never enter the id check and a collision between two rotating hazards would not be +/// reported. The ids are recoverable statically -- the selector is an N-way +/// `arith.select` chain over constants -- so closing this is mechanical. + +/// Number of Error-severity violations (warnings do not fail the gate). +unsigned countErrors(llvm::ArrayRef violations); + +void printInterferenceViolations( + llvm::raw_ostream &os, llvm::StringRef funcName, llvm::StringRef view, + size_t recordCount, llvm::ArrayRef violations); + +//===----------------------------------------------------------------------===// +// The happens-before edge model +//===----------------------------------------------------------------------===// +// +// `computeCoverage` derives the orderings a function's emitted sync establishes. +// G1-self consumes it: a dependency is covered when the model orders its endpoints. +// +// Anchors are the ops carrying `OpPipeInterface` and a read/write memory effect, +// numbered in program order; sync ops are never anchors. An edge (i, j) means the +// emitted sync orders anchor i before anchor j, and edges come only from emitted +// sync: +// +// barrier at slot s => all i < s, all j >= s +// barrier at slot s => i < s on pipe X, j >= s on pipe X +// set_flag(P,Q,e)@s / wait@w => i < s on pipe P, j >= w on pipe Q +// rls_buf(b)@r on P, get_buf(b)@g>r on Q +// => i < r on pipe P, j >= g on pipe Q +// +// The last rule is the buffer token's happens-before: `get_buf` acquires what the +// previous holder released. Without it a buffer-id allocation appears to order +// nothing and every dependency on it reads as uncovered. +// +// Program order is NOT an edge, not even within one pipe: `pto.barrier` is an +// intra-pipeline barrier, so crediting program order would credit orderings nothing +// established. Two exceptions, both narrow: +// - two accesses to the SAME tile buffer are ordered by program order, so nothing +// is owed for such a pair on any arch; +// - A5 additionally guarantees PIPE_V intra-pipe ordering, which is why +// `SyncCodegen::CreateBarrierOp` emits nothing for a V->V pair there and the +// backend rejects one. A2 and A3 have no such guarantee. +// +//===----------------------------------------------------------------------===// +// THE ITERATION DIMENSION -- loop-carried (backward) orderings. +//===----------------------------------------------------------------------===// +// +// The forward model is a flat sequence, so "iteration k before iteration k+1" is +// inexpressible in it, not merely filtered by the `i < j` guard. A second, +// distance-annotated edge class is therefore carried alongside: +// +// CarriedEdge{from=j, to=i, distance=1} +// "anchor j in iteration k happens-before anchor i in iteration k+1" +// +// Three rules build it, kept strictly separate: +// +// RULE 1 (detect). Within ONE loop body, match set/wait (and rls/get) forward +// per key; whatever is LEFT OVER wraps, so an unmatched in-body set pairs with an +// unmatched in-body wait at a lower slot. The head/tail compensation ops take no +// part -- they sit outside the body, so a correct kernel and one missing its +// compensation are detected identically. The forward pass already consumes the +// compensation as two forward pairs, which is why bidirectional pairing alone +// would establish nothing here. +// +// RULE 2 (generate). For a carried pair (set@p, wait@q), emit (x, y, d=1) for +// body anchors x < p on the src pipe and y >= q on the dst pipe. BOTH endpoints +// are restricted to the carrying loop's body: a prologue anchor has no +// iteration index, so a d=1 edge touching one would be meaningless. +// +// RULE 3 (credit). An EVENT carried pair earns its edges only if PRIMED: +// P1' a set_flag H on the same (srcPipe, dstPipe, id) lies in a block on L's +// ANCESTOR SPINE -- the block holding L, or the block holding one of L's +// ancestor ops. REACHABILITY, not nesting: a head under an `scf.if`, or in +// a possibly-zero-trip sibling loop, may never run, and `scf.if` is not a +// LoopLikeOpInterface so a nesting test cannot see it. +// P2 H precedes L in program order. +// P3 H is still live at loop entry -- no wait_flag on the same key lies +// outside L between H and L. +// An unprimed backward pair establishes nothing: iteration 1's wait blocks +// forever. Rule 1 alone cannot tell a correct kernel from a deadlocking one, +// since both carry the in-body pair; credit is what discriminates. +// +// RULE 3b. The same reachability test on the in-body side: the carried producer, +// consumer and any covering barrier must be DIRECT children of L's body, or +// iterations 2..N have nothing to wait on. +// +// BARRIER COVERAGE. A candidate may serialize instead of pairing, leaving Rule 1 +// no pair to find. A barrier at body slot `s` covers (x, y, d=1) iff +// `x < s || y >= s`; the only uncovered case is the wrap gap `y < s <= x`. +// +// BUFFER TOKENS ARE EXEMPT FROM RULE 3: the ticket lock starts free, so +// iteration 1's `get_buf` needs no priming op. +// +// Only the HEAD set is required, not the tail wait. A missing drain is a leaked +// flag, which is G2's `event-id-unclosed`; demanding it here would duplicate G2. +// +// KNOWN GAPS. None can produce a false PASS -- each is either a false FAILURE on a +// shape no in-tree mode emits, or a weakening shared by both sides: +// - PRIME ID MISMATCH: a compensation priming with a different id than the +// in-body pair loses credit. Both share one id today; rotation must defuse it. +// - PEELED FIRST ITERATION: needs no priming, but P1' still demands one. Nothing +// in-tree peels. A guarded PRIME is a different shape and is correctly rejected. +// - LIVENESS IS A LINEAR SCAN, not dataflow: a consuming wait inside a not-taken +// `scf.if` still counts, so credit can be withheld from a primed pair. +// Conservative by construction. +// - NESTED-LOOP PRODUCER: a carried pair whose producer sits in a nested loop is +// attributed to that loop, so an outer stranded consumer finds no partner. Both +// sides lose it equally, so it weakens the gate rather than inverting a verdict. +// +// ROTATING PAIRS are modelled, in `computeCoverage`, on both sides: a dyn pair +// contributes a CARRIED edge at distance N recovered from the selector's own +// `arith.remui` modulus, and no forward edge (within one iteration the set and wait +// touch different ids). Demand is likewise at distance N, from +// `baseAddresses.size()`, because with N slots iteration k+1 touches a different +// slot and the real conflict is at reuse. WIDEN, NEVER DROP: the forward precedent +// `isForwardDepDroppableBySlotAffine` drops a slot-disjoint dep, which is wrong on +// the back edge -- it would excuse the dependency and let an unrotated kernel pass. +// The closure is capped at 32; above the cap a composition falls to UNCOVERED. + +struct CarriedEdge { + unsigned from = 0; + unsigned to = 0; + unsigned distance = 1; + bool operator<(const CarriedEdge &o) const { + return std::tie(from, to, distance) < std::tie(o.from, o.to, o.distance); + } + bool operator==(const CarriedEdge &o) const { + return from == o.from && to == o.to && distance == o.distance; + } +}; + +struct CoverageProfile { + unsigned anchorCount = 0; + /// Sorted, unique. `first < second`, both anchor indices. + llvm::SmallVector> edges; + /// Sorted, unique. Loop-carried orderings; `from`/`to` are the same global + /// anchor indices as `edges`, and `from < to` is NOT implied. + llvm::SmallVector carriedEdges; + /// Hash of the anchor sequence. A differing signature means the two runs did not + /// compile the same program, so their edge sets are not comparable. + /// + /// Must be computed with a digest that is STABLE ACROSS PROCESSES, since the value is + /// written to a file by one ptoas run and compared by another. `llvm::hash_value` is + /// not: under LLVM_ENABLE_ABI_BREAKING_CHECKS its seed is the address of a function, + /// so ASLR changes it every execution and identical source compares unequal. + uint64_t anchorSignature = 0; + + /// Which digest produced `anchorSignature`. A profile from an older build parses this + /// as 0 and is rejected as stale rather than silently compared against a value + /// computed a different way. + unsigned signatureVersion = 0; + /// PIPE_ALL barriers seen. Reported by the differential gate and never + /// differenced into its verdict: a reference that discharges hazards with + /// barriers is denser by construction, and so is a candidate that is simply + /// broken. + unsigned barrierAll = 0; +}; + +/// Walks `func` and derives the happens-before edges its emitted sync induces. +/// +/// `anchorIndex`, when non-null, receives the Operation* -> anchor-index map +/// built by the same walk. G1-self needs it to map a dependency's endpoints into +/// this coordinate system, and taking it from here rather than rebuilding it is +/// what guarantees the two views cannot drift. +CoverageProfile +computeCoverage(func::FuncOp func, + llvm::DenseMap *anchorIndex = nullptr); + +enum class CoverageViolationKind { + MissingOrdering, + MissingBackwardOrdering, + AnchorMismatch, + MissingReference, + /// The persisted profile was written by a build whose signature digest differs from + /// this one's. Reported SEPARATELY from AnchorMismatch on purpose: the profile is + /// stale, not the program, and telling the two apart is the difference between + /// "regenerate the profile" and "your kernel changed". + ProfileVersionMismatch, +}; + +/// Bumped whenever the digest behind `CoverageProfile::anchorSignature` changes, so a +/// profile written by an older build is refused instead of compared. Version 1 was +/// `llvm::hash_value`, which is not stable across processes; version 2 is xxh3. +constexpr unsigned kCoverageSignatureVersion = 2; + +llvm::StringRef coverageViolationKindName(CoverageViolationKind kind); + +struct CoverageViolation { + CoverageViolationKind kind; + unsigned from; + unsigned to; + std::string detail; + /// Only meaningful for MissingBackwardOrdering; 0 for the forward class. + unsigned distance = 0; +}; + + +/// G1: every ordering in `reference` must also be in `candidate`. +llvm::SmallVector +checkCoverageSuperset(const CoverageProfile &candidate, + const CoverageProfile &reference); + +void printCoverage(llvm::raw_ostream &os, llvm::StringRef funcName, + const CoverageProfile &profile); + +llvm::StringMap parseCoverage(llvm::StringRef text); + +/// Prints at most 8 missing orderings, a fixed cap with no parameter and no flag to +/// raise it. On a large failure the survivors are the numerically first anchor pairs, +/// which is the least informative slice available -- attributing a cause means dumping +/// both profiles and differencing them, not reading this report. +void printCoverageViolations(llvm::raw_ostream &os, llvm::StringRef funcName, + const CoverageProfile &candidate, + const CoverageProfile &reference, + llvm::ArrayRef violations); + +//===----------------------------------------------------------------------===// +// G1-self: ABSOLUTE coverage against the dependency analysis +//===----------------------------------------------------------------------===// +// +// G1 proper is DIFFERENTIAL: it compares the candidate against the reference's +// coverage, so a green result can only ever mean "matches the reference", and its +// strength is bounded by that reference. G1-self is ABSOLUTE: it asserts that +// every dependency the FRONT-END dependency analysis reports has an emitted sync +// ordering its source before its destination. +// +// WHERE THE EXPECTED SET COMES FROM, and why it is not circular. The expected set +// is built by running `PTOIRTranslator` (def/use extraction and pipe assignment) +// and then calling `MemoryDependentAnalyzer::DepBetween` over the resulting +// `CompoundInstanceElement::defVec` / `useVec` DIRECTLY. Nothing in it reads +// InsertSync's decisions: not which hazards it chose to sync, not the mechanism +// it picked, not the ids it allocated, not its emitted ops. In particular this +// check deliberately does NOT call `InsertSyncAnalysis::IsMemInfoHasDependency`, +// which layers InsertSync's POLICY on top of DepBetween (the tload->tload WAW +// exemption, the ACC read/read special case). Policy is what this gate audits, +// so it cannot be part of the oracle. +// +// HONEST LIMIT OF "ABSOLUTE". G1-self shares the front end -- def/use +// extraction and the alias analysis -- with InsertSync, because that IS the +// dependency side. So it cannot catch a dependency the front end itself fails to +// see (an alias `MemAlias` misses). It is independent of InsertSync's SYNC +// decisions, which is what makes it non-circular; it is not independent of the +// memory model. Claim accordingly. +// +// SELF-PAIRS ARE ENUMERATED SEPARATELY. The main pair loop is `for j = i + 1`, so +// it never pairs a compound with itself; a same-op loop-carried hazard -- op x in +// iteration k against the same op in k+1 -- is asked about by a second loop, which +// records the three hazard classes as `waw`/`raw`/`war/carried-self`. Only ops +// inside a loop are considered: without a back edge there is no second execution. +// +// Coverage answers these without special-casing. A same-pipe body barrier records a +// carried edge over every (x, y) anchor pair in its loop span subject to +// `x < slot || y >= slot`, which is trivially true when x == y, so the barrier +// contributes a carried SELF-edge at distance 1. Nothing else seeds the closure +// diagonal -- every cell starts at `kUnreached` -- so a self-pair is covered +// exactly when some emitted op orders the op against its own next execution. +// +// ONE LIMIT ON THAT, and it matters when reading a green result: coverage is keyed +// on the anchor pair alone, with no loop-nest attribution. A distance-1 self-edge +// contributed by an ENCLOSING loop's back edge therefore credits a dependence +// carried by an inner loop, so the diagonal can read covered whether or not +// anything orders the op one nest level down. +// +// COVERED, per mechanism. The covering relation is `c subset-of h`: the sync's +// interval must be CONTAINED in the dependency's. Reusing `computeCoverage` +// gives exactly that for all three mechanisms, which is the main reason to reuse +// it rather than re-derive: +// event set@s / wait@w => edge (i, j) for i < s, j >= w. So the edge +// exists iff [s, w) lies inside (i, j] -- containment, not mere +// co-occurrence. +// barrier PIPE_ALL at s => edge (i, j) for i < s <= j: a barrier strictly +// between the two endpoints. +// buffer-ID rls(b)@r, get(b)@g>r => edge (i, j) for i < r, j >= g. This is the +// TOKEN PROTOCOL -- release before acquire ON THE SAME buf_id -- not +// co-location. A check blind to the protocol would bless a +// split-endpoint hazard that is silently unordered. +// +// TRANSITIVITY IS TAKEN, and it must be. Happens-before is transitive, but +// `computeCoverage`'s edge set is not closed: pipe filtering means a chain +// a(PIPE_A) -> b(PIPE_B) -> c(PIPE_C) is covered by two pair-wise syncs while the +// direct edge (a, c) is absent, since neither sync matches both endpoints' pipes. +// Checking against the raw set would report a false "uncovered" there. So the +// relation is closed before checking, with distances composing ADDITIVELY +// (0+0=0, 0+1=1, 1+1=2, ...). A composition is dropped only when the sum exceeds +// `kMaxDistance` (32), leaving the pair unreached, which is the conservative +// direction. +// +// THE MAPPING, and what happens to what it cannot map. Dependencies live over +// `BaseMemInfo`, which carries no `Operation*`; coverage lives over anchor +// indices. The bridge is `CompoundInstanceElement::elementOp` -> the anchor index +// assigned by `computeCoverage`'s walk. A dependency whose endpoint op is not an +// anchor (no `OpPipeInterface`, or no read/write memory effect) CANNOT be +// expressed in the coverage coordinate system. Those are counted and reported as +// `unmappable`, never silently dropped -- silent drops are exactly how this check +// would become vacuous. + +struct SelfCoverageViolation { + unsigned srcAnchor; + unsigned dstAnchor; + unsigned distance; // 0 = same iteration, 1 = loop-carried + std::string hazard; // raw | war | waw + std::string srcName; + std::string dstName; + std::string srcPipe; + std::string dstPipe; +}; + +struct SelfCoverageReport { + unsigned dependencies = 0; // total requirements derived from DepBetween + unsigned covered = 0; + unsigned carriedDeps = 0; // of those, how many are loop-carried (d=1) + unsigned carriedCovered = 0; + unsigned unmappable = 0; // endpoint has no anchor representation + unsigned armExcluded = 0; // (a2) pairs on mutually exclusive scf.if arms + /// (a1) A5 PIPE_V->PIPE_V: ordered by the target guarantee the codegen relies + /// on when it declines to emit a barrier. NOT "A5 same-pipe". + unsigned archGuaranteed = 0; + /// (a3) PIPE_S->PIPE_S. Counted apart from `archGuaranteed` because no hardware + /// guarantee is claimed for it -- see the exemption site for what it does rest on. + unsigned pipeSelfOrdered = 0; + unsigned uncoveredSamePipe = 0; // diagnosis split: both endpoints one pipe + unsigned uncoveredCrossPipe = 0; // the interesting ones + unsigned compounds = 0; // SyncIR compounds seen (consistency check) + llvm::SmallVector violations; +}; + +/// G1-self: every dependency `DepBetween` reports must be ordered by emitted sync. +SelfCoverageReport computeSelfCoverage(func::FuncOp func); + +/// `cap` bounds the violations printed per function; 0 prints all. A capped report +/// HIDES CLASSES -- two cross-pipe carried outliers on `rope_kv_cache` sat past the +/// default cap and went undiagnosed through a whole round of analysis -- so raise it +/// when attributing causes. +void printSelfCoverage(llvm::raw_ostream &os, llvm::StringRef funcName, + const SelfCoverageReport &report, unsigned cap = 8); + +} // namespace oracle +} // namespace pto +} // namespace mlir + +#endif // MLIR_DIALECT_PTO_TRANSFORMS_INSERTSYNC_SYNCORACLEGATES_H diff --git a/include/PTO/Transforms/InsertSync/UnifiedSyncModel.h b/include/PTO/Transforms/InsertSync/UnifiedSyncModel.h new file mode 100644 index 0000000000..c88dc95908 --- /dev/null +++ b/include/PTO/Transforms/InsertSync/UnifiedSyncModel.h @@ -0,0 +1,605 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===- UnifiedSyncModel.h - Unified sync allocator data model -------------===// +// +// Event ids, buffer ids and barriers are three resource classes over one +// interval-coloring problem. This file states that problem -- the hazard, its +// interval, the per-class resource model, the per-hazard cost -- and decides +// nothing. Allocation, routing and codegen live in the pass. +// +//===----------------------------------------------------------------------===// + +#ifndef PTO_TRANSFORMS_INSERTSYNC_UNIFIEDSYNCMODEL_H +#define PTO_TRANSFORMS_INSERTSYNC_UNIFIEDSYNCMODEL_H + +#include "PTO/Transforms/InsertSync/SyncCommon.h" +#include "llvm/ADT/DenseMap.h" +#include "llvm/ADT/SmallVector.h" +#include "llvm/Support/raw_ostream.h" + +namespace mlir { +namespace pto { +namespace unified { + +//===----------------------------------------------------------------------===// +// Joining a hazard to its buffer-id cluster +//===----------------------------------------------------------------------===// + +/// Which buffer-id cluster(s) a dependency's memory belongs to. +/// +/// Built by the caller: deciding that two tiles are the same buffer needs +/// `MemoryDependentAnalyzer`, which this header must not depend on. +/// +/// Cluster identity is `MemAlias`, not a tuple. Three near-misses this must not be +/// simplified back to: `BaseMemInfo::rootBuffer` is the root ALLOCATION, coarser +/// than a buffer, so it conflates tiles that do not alias; `TileInfo::rootBuffer` +/// never matches, because BufidSync rewrites it to the tile value for a +/// `ConstantLike` root; `(scope, baseAddr, size)` degenerates before addresses are +/// assigned, when `baseAddresses` is empty and every `baseAddr` collapses to 0. +using MemInfoToClusters = + llvm::DenseMap>; + +//===----------------------------------------------------------------------===// +// Interval +//===----------------------------------------------------------------------===// + +/// A hazard's live range in SyncIR-index space -- deliberately not op position. +/// MoveSyncState hoists a sync by rewriting these indices without moving an +/// operation, which is what lets event and buffer-ID hazards coexist: hoisting +/// relocates the interval while a buffer-ID hazard's op anchor is untouched. +/// +/// Two hazards may share a physical id only if their intervals are disjoint. +/// Touching endpoints do NOT overlap: an id freed at index i is reusable at i. +struct Interval { + unsigned start = 0; ///< SyncIR index of the set/acquire side. + unsigned end = 0; ///< SyncIR index of the wait/release side. + + bool overlaps(const Interval &o) const { + return start < o.end && o.start < end; + } +}; + +//===----------------------------------------------------------------------===// +// Hazard -- the unit of allocation, and the unit of MECHANISM +//===----------------------------------------------------------------------===// + +/// One ordering to enforce: producer on `srcPipe` before consumer on `dstPipe`. +/// Owns both halves of the sync pair. +/// +/// THE BOTH-HALVES INVARIANT. The mechanism belongs to the hazard, not to an +/// individual SyncOperation, and `SetMechanism` is the only writer. `GetMatchSync` +/// does copy the mechanism to the half it creates, but it built this pair inside +/// InsertSyncAnalysis, before any mechanism was chosen, so a later stamp on one +/// half leaves the other on its type-derived default. A set on one mechanism whose +/// wait is on another is not a pair but two unrelated syncs, and no gate catches +/// it: G1 sees the ordering, G2 sees no id conflict, G4 sees the same op count. +class Hazard { +public: + using MECHANISM = SyncOperation::MECHANISM; + + Hazard(unsigned index, SyncOperation *setOp, SyncOperation *waitOp) + : index_(index), setOp_(setOp), waitOp_(waitOp) {} + + unsigned index() const { return index_; } + SyncOperation *setOp() const { return setOp_; } + SyncOperation *waitOp() const { return waitOp_; } + + /// Whether a wait side ever existed -- STRUCTURE, not mechanism. + /// `SpillToBarrier` converts a pair without dropping its wait object, so a + /// spilled hazard reports `isBarrier() == false` and `mechanism() == BARRIER`. + /// Both readings are needed: the colourer skips structurally-lone barriers + /// because there is no pair to colour, while codegen and the gates need the + /// spilled pair's mechanism. "Is this realised as a barrier" is `mechanism()`. + bool isBarrier() const { return waitOp_ == nullptr; } + + PipelineType srcPipe() const { return setOp_->GetActualSrcPipe(); } + PipelineType dstPipe() const { return setOp_->GetActualDstPipe(); } + + const Interval &interval() const { return interval_; } + void setInterval(Interval iv) { interval_ = iv; } + + /// How this hazard is realised. Reads the set side; the both-halves invariant + /// keeps the wait side in step. Can disagree with `isBarrier()` -- see there. + MECHANISM mechanism() const { return setOp_->GetMechanism(); } + + /// Route to the buffer-ID token protocol, so this hazard must NOT take an event + /// id. + /// + /// THE `uselessSync` MARKS ARE NOT OPTIONAL. `SyncCodegen` dispatches on the + /// SyncOperation TYPE, never on the mechanism, so stamping BUFID alone would + /// still emit the set/wait pair, now with no event id. All four ops, because + /// `SetEventIds` stamps head/tail too and those are real ops in syncIR's pipe + /// lists; a token needs no priming pair, its counter starts free. + void RouteToBufid() { + SetMechanism(MECHANISM::BUFID); + setOp_->uselessSync = true; + if (waitOp_) + waitOp_->uselessSync = true; + if (headOp_) + headOp_->uselessSync = true; + if (tailOp_) + tailOp_->uselessSync = true; + } + + /// Realise as a PIPE_ALL barrier when the colourer cannot serve the hazard. "No + /// id" is not a resting state: codegen emits the pair regardless, so an unserved + /// hazard would go out carrying id 0 or the unset sentinel. + /// + /// THE WAIT SIDE IS THE ONE KEPT. A PIPE_ALL at the wait position sits before the + /// consumer; for a backward hazard the in-body wait is at the top of the body, so + /// the barrier drains the previous iteration, which is exactly the carried edge. + /// At the set side it would sit after the producer and order nothing across the + /// back edge. Head and tail are suppressed for the reason given in + /// `RouteToBufid`, and a PIPE_ALL needs no priming anyway. + void SpillToBarrier() { + if (!waitOp_) + return; // already a barrier; the colourer never reaches these + setOp_->SetPipeAll(); + waitOp_->SetPipeAll(); + setOp_->uselessSync = true; + if (headOp_) { + headOp_->SetPipeAll(); + headOp_->uselessSync = true; + } + if (tailOp_) { + tailOp_->SetPipeAll(); + tailOp_->uselessSync = true; + } + } + + /// THE ONLY WAY a mechanism may be assigned. Stamps both halves. + void SetMechanism(MECHANISM m) { + setOp_->SetMechanism(m); + if (waitOp_) + waitOp_->SetMechanism(m); + } + + /// How many event ids must be live across this hazard's interval. 1 ordinarily; + /// N for a multi-buffered (rotating) pair, where the front end supplies a slot + /// SSA via `pto.multi_tile_get %mb[%slot]` and `eventIdNum = N`, and codegen + /// emits `set_flag_dyn`/`wait_flag_dyn` selecting `eventIds[slot % N]`. Read + /// from the set side; the both-halves invariant keeps the two in step. + unsigned demand() const { + return std::max(1u, static_cast(setOp_->eventIdNum)); + } + + /// Assign ids to every op the hazard owns, from ONE coloring decision: set, wait, + /// and for a loop-carried hazard the synthesised head-set and tail-wait too. + /// Stamping them together is what makes the head prime and the tail drain the + /// same flag the in-body pair uses. + /// + /// Arity matters more than identity here: `CreateSetWaitOpForMultiBuffer` asserts + /// `eventIds.size() == slotCount` and indexes `eventIds[i]`, so a short list is + /// an out-of-bounds read rather than a mis-sync. + void SetEventIds(llvm::ArrayRef ids) { + assert(!ids.empty() && "a hazard must get at least one id"); + setOp_->eventIds.assign(ids.begin(), ids.end()); + if (waitOp_) + waitOp_->eventIds.assign(ids.begin(), ids.end()); + if (headOp_) + headOp_->eventIds.assign(ids.begin(), ids.end()); + if (tailOp_) + tailOp_->eventIds.assign(ids.begin(), ids.end()); + } + + void SetEventId(int id) { + setOp_->eventIds.assign(1, id); + if (waitOp_) + waitOp_->eventIds.assign(1, id); + if (headOp_) + headOp_->eventIds.assign(1, id); + if (tailOp_) + tailOp_->eventIds.assign(1, id); + } + bool hasEventId() const { return !setOp_->eventIds.empty(); } + + //-- Loop-carried compensation ----------------------------------------------- + // + // A loop-carried hazard's in-body pair (wait, then set) deadlocks on iteration 1 + // with nothing priming the flag. `synthesizeLoopCompensation` adds a head-set + // before the carrying loop and a tail-wait after it, on the same id. The hazard + // holds raw pointers to them; both null for a forward hazard. + + /// Attach the synthesised head-set / tail-wait. Set once, before coloring. + void setCompensation(SyncOperation *head, SyncOperation *tail) { + headOp_ = head; + tailOp_ = tail; + } + SyncOperation *headOp() const { return headOp_; } + SyncOperation *tailOp() const { return tailOp_; } + bool hasCompensation() const { return headOp_ != nullptr; } + + //-- Buffer-ID cost is reuse-conditioned ------------------------------------- + // + // A token's get/rel counters are bidirectional: ONE token discharges the forward + // RAW and the reverse WAR. So a token beats an event only where the buffer really + // is reused in both directions; forward-only it adds a reverse edge nobody asked + // for. The cost needs the buffer's whole hazard group, not the hazard alone -- + // hence these two fields. + + /// Every buffer-id cluster this hazard's memory belongs to. Resolved per + /// `BaseMemInfo *`, by pointer identity with a `MemAlias` fallback; empty when + /// none of the hazard's tiles is buffer-id-clusterable. + llvm::SmallVector bufferClusters; + + /// Does a mirror-direction hazard share a cluster with this one? + /// + /// CLUSTERS, NOT `depRootBuffers`. A root is the root ALLOCATION, so keying on it + /// reports two hazards on different tiles as reverse partners merely because they + /// share a root. That false positive is the expensive direction: it prices a token + /// low and routes a forward-only hazard onto it. + bool hasReverseWar = false; + + + /// Can this hazard be realised as a buffer-id token AT ALL? Routing must refuse a + /// buffer carrying any hazard for which this is false: suppressing the event ops + /// of a hazard no token can express loses the ordering outright. + /// + /// Four ways a hazard is inexpressible, each because the token names something + /// different from what the hazard needs: + /// - SAME-PIPE. A token orders accesses to one buffer; ordering a pipe against + /// itself needs a barrier, which is a different class. + /// - NO CLUSTER. With no aliasing clique there is no token to take. + /// - GM, or endpoints in DIFFERENT scopes. A token brackets a tile buffer; + /// `BufidSyncAnalysis` excludes both shapes for the same reason. + /// - A PIPE THE EMITTER CANNOT ENCODE. `BufidSyncCodegen` maps a pipe to a + /// `SyncOpType` and fails the compilation on the ones it has no name for. + bool tokenExpressible = false; + + /// Why this hazard's two endpoints sit at one address, when they do. + /// + /// A hazard whose endpoints are DISTINCT allocations at the same address in the + /// same scope is ordered because the addresses coincide: no SSA value flows + /// between two distinct allocations, so the ordering is invisible as dataflow. + /// Classified by whether those allocations also share a tile type: + /// - `Reuse` -- same type, one buffer holding successive values; + /// - `AliasView` -- different types over the same bytes, a reshape or + /// transpose view of live data. + /// + /// INFORMATIONAL ONLY. No routing, refusal, mechanism choice or emission may read + /// this. The type test is a heuristic: two same-typed tiles can be aliased on + /// purpose, and the ordering between them is then a real read-after-write through + /// memory. A decision taken on a wrong classification would drop a live ordering. + enum class AddressSharing { + None, ///< one allocation on both sides, or the addresses differ + Reuse, ///< distinct allocations, one address, SAME tile type + AliasView, ///< distinct allocations, one address, DIFFERING tile types + }; + AddressSharing addressSharing = AddressSharing::None; + + /// The shared slot and the two allocations in program order. Meaningful only when + /// `addressSharing != None`. + uint64_t sharedAddress = 0; + pto::AddressSpace sharedScope = pto::AddressSpace::Zero; + Operation *sharedAllocFirst = nullptr; + Operation *sharedAllocSecond = nullptr; + +private: + unsigned index_; + SyncOperation *setOp_; ///< never null + SyncOperation *waitOp_; ///< null for a barrier + SyncOperation *headOp_ = nullptr; ///< synthesised head-set; null unless loop-carried + SyncOperation *tailOp_ = nullptr; ///< synthesised tail-wait; null unless loop-carried + Interval interval_; +}; + +//===----------------------------------------------------------------------===// +// Per-class resource model +//===----------------------------------------------------------------------===// + +/// The three id pools, with no arch branch anywhere. +/// +/// K IS THE ARCHITECTURE. K=0 leaves the buffer-ID class empty and the allocator +/// degenerates to event+barrier, which is A3; K=32 opens it, which is A5. There is +/// no `if (arch == a5)` in the allocator and there must never be one. +struct ResourceModel { + /// Buffer-ID pool: ONE pool, shared across all pipes and both directions. + unsigned bufidCapacity = 0; // K + + /// Event pool for one direction. Event flags are per-(src,dst) DISJOINT hardware + /// -- id 0 in MTE2->V is a different flag from id 0 in V->MTE3 -- so this is a + /// pool per direction, not a shared pool of 8. Some directions (V<->S) reserve + /// their top id, so it is not always 8. + /// + /// Reads the same `SyncEventIdAllocation::GetReservedEventIdNum` the G3 gate + /// reads, so allocator and gate cannot disagree about the bound. + unsigned eventPoolSize(PipelineType src, PipelineType dst) const; + + /// Barrier is the spill class: unbounded and always available. That is what makes + /// the allocator total -- there is no "no id left" failure mode, only a worse + /// schedule. + static constexpr bool barrierIsUnbounded = true; + +}; + +//===----------------------------------------------------------------------===// +// alpha -- the per-hazard mechanism cost +//===----------------------------------------------------------------------===// + +/// What a hazard costs on each resource class. +/// +/// SHAPE ONLY: an ordering, not a calibrated model. No allocation decision reads +/// alpha -- `routeBuffers` tests `Buffer::isWrittenBack` directly, and only +/// `printSyncModel` renders these numbers. +struct Alpha { + unsigned event = 1; + unsigned bufid = 1; + unsigned barrier = 100; ///< spill: serializes the core, so it must lose to any id. + + /// A token is cheap only when its reverse WAR is really wanted; forward-only it + /// adds a spurious back edge, so it is priced above an event. + static Alpha forHazard(const Hazard &h) { + Alpha a; + a.bufid = h.hasReverseWar ? 1 : 2; + return a; + } +}; + +//===----------------------------------------------------------------------===// +// Buffer -- the unit buffer-ID routing decides on +//===----------------------------------------------------------------------===// +// +// THE BUFFER, NOT THE HAZARD, IS THE ROUTING UNIT, and that is correctness rather +// than a modelling preference. The protocol is a COUNTER CYCLE: every `get_buf` is +// matched by an `rls_buf` on the same id. If some hazards on one buffer are +// realised as events while others take get/rls, the cycle is missing a step and the +// acquire blocks forever. So routing is decided per buffer, before any hazard on it +// takes an event id. +// +// The reservation window is the UNION SPAN, matching +// `BufidSyncIdAlloc::computeLifeIntervals`: an access outside a loop contributes +// its own `syncIRIndex`, one inside contributes the whole `[loopBegin, loopEnd]`, +// and the result is min-start/max-end per logic id. +// +// Over-reserving is WASTEFUL, NOT WRONG, which is why a buffer with no single loop +// scope gets a wide window instead of being refused. The get_buf/rls_buf ops are +// emitted at the access sites regardless, and the window only decides which +// physical id a logic id maps to, so a window that is too long can lose an id but +// never an ordering -- and the exhaustion path coalesces logic ids rather than +// failing. +struct Buffer { + /// The aliasing-clique id (`bufferClusters` entry). One token covers one clique. + int logicId = -1; + + /// Does anything write this buffer back? + /// + /// NOT THE SAME PREDICATE AS `Hazard::hasReverseWar`, and the difference is + /// deliberate. Per-hazard asks whether a mirror-direction hazard shares a cluster + /// with one hazard; per-buffer asks whether the buffer carries a mirror pair at + /// all, so a third hazard in an unmirrored direction is true here and false + /// there. Per-buffer is the correct one: the token is a property of the buffer, + /// and since routing is all-or-nothing a per-hazard predicate could let two + /// hazards on one buffer disagree, which is the split that hangs. + bool isWrittenBack = false; + + /// Union of the hazards' own intervals, in SyncIR-index space: a forward hazard + /// contributes its tight [set, wait) range, a loop-carried one its whole carrying + /// loop. `bufidLoop` is the reservation derived from these. + Interval accessRange{0, 0}; + + /// Any access inside a loop. + bool isLoopCarried = false; + + /// Accesses live in more than one loop context (inside and outside, or in two + /// different loops), so there is no single well-defined carrying loop. + bool spansLoopBoundary = false; + + /// How many distinct loop contexts the accesses occupy (function level counts + /// as one). 1 means a single well-defined scope. + unsigned loopContextCount = 0; + + /// The reservation window. Union span -- an access outside a loop contributes + /// itself, an access inside contributes its whole loop. + Interval bufidLoop{0, 0}; + + /// Routing outcome. `true` means every hazard touching this buffer is to be + /// realised as a buffer-ID token, and NONE of them may take an event id. + bool routedToBufid = false; + + /// Peak per-direction overlap predicted for the directions this buffer touches + /// (the maximum over them), and the event pool of the direction that peak came + /// from -- not the minimum pool. Routing compares these; recorded so the decision + /// is auditable rather than a bare bool. + unsigned predictedPeakOverlap = 0; + unsigned smallestPool = 0; + + /// False means the reservation window could not be resolved. Such buffers are + /// COUNTED in the report rather than defaulted to a plausible-looking interval: a + /// wrong window is a wrong reservation, which is a missing get/rls, which is a + /// hang. + bool bufidLoopDefined = false; +}; + +/// A (direction, id) pair the pto-isa library occupies INSIDE a macro call, over +/// that call's span. The id is not chosen here, it is already taken, so the +/// colourer must treat it as occupancy rather than as a candidate. +struct HiddenReservation { + int srcPipe = -1; + int dstPipe = -1; + int id = -1; + Interval span; ///< SyncIR indices, already padded + llvm::StringRef macroName; ///< for diagnostics only +}; + +/// The hazard set, plus the resources it must be colored into. +struct SyncModel { + llvm::SmallVector hazards; + /// One entry per buffer-id cluster touched by any hazard, sorted by logicId. + llvm::SmallVector buffers; + ResourceModel resources; + /// Ids already spoken for by a macro's library implementation. Empty on any + /// kernel with no macro ops, which is why this costs nothing to consult. + llvm::SmallVector hiddenReservations; +}; + +/// Build the model from the reused front end's output. `SyncOperations`' OUTER +/// index is already the hazard -- one group per ordering, holding its set and its +/// wait -- which is the granularity the both-halves invariant needs, and why +/// `Hazard` can own the pair without re-deriving the pairing. +/// +/// `memInfoToClusters` joins a hazard's memory to its cluster(s); an empty map is +/// legal and simply leaves every hazard unclustered with `hasReverseWar` false. +/// +/// Assigns no ids, emits no IR, and does not mutate the SyncOperations. `syncIR` is +/// read only to resolve a loop-carried hazard's CARRYING loop: +/// `setOp->GetForEndIndex()` indexes a `LoopInstanceElement` whose +/// `[beginId, endId + 1)` is that hazard's interval -- the extra index covers the +/// tail-wait that sits at endId -- read forward, never inverted. +SyncModel buildSyncModel(const SyncOperations &syncOps, unsigned bufidCapacity, + const MemInfoToClusters &memInfoToClusters, + const SyncIRs &syncIR); + +/// Synthesise the head-set / tail-wait pair for every loop-carried +/// (`GetForEndIndex`) hazard, so its in-body backward pair is primed before the +/// carrying loop and drained after. Without them the kernel deadlocks on iteration +/// 1; this is not an optimisation. Returns the number of pairs synthesised. +/// +/// Appends ONE `SyncOperations` group per such hazard and links both ops onto the +/// hazard, so one `SetEventIds` stamps all four with one id. Exactly one head and +/// one tail per hazard. +/// +/// Also PLACES each pair into `syncIR` -- head into the `pipeBefore` of the anchor +/// `hoistAnchorFor` picks (the carrying loop's begin, or an earlier op in that +/// loop's own enclosing region when one is legal), tail into the loop end's +/// `pipeAfter`, with `push_front` so the +/// tail precedes any existing loop-end set. `SyncCodegen` walks those lists, so +/// owning the pair in `syncOps` alone would let emission drop it silently. Hence +/// `syncIR` is mutable here. +/// +/// Runs after `buildSyncModel` and before `colorEventIds`. +unsigned synthesizeLoopCompensation(SyncModel &model, SyncOperations &syncOps, + SyncIRs &syncIR); + +/// Deterministic, diff-friendly rendering (one hazard per line). +void printSyncModel(llvm::raw_ostream &os, llvm::StringRef funcName, + const SyncModel &model); + +//===----------------------------------------------------------------------===// +// Event interval-coloring allocator +//===----------------------------------------------------------------------===// + +/// Result of the event-id coloring. +/// +/// Two conventions that read wrong at d > 1: `reused` counts ID ASSIGNMENTS, so one +/// hazard of demand d can add up to d, and `assigned` is per-HAZARD while +/// `idsAssigned` is per-ID. +struct EventColorResult { + /// Hazards the colourer could not serve, realised as PIPE_ALL + /// barriers. Equal to `overflow` -- kept separate so a future partial spill is + /// visible rather than silently folded into the overflow count. + unsigned spilledToBarrier = 0; + unsigned assigned = 0; ///< hazards that got an event id + unsigned overflow = 0; ///< hazards whose direction had no free id at that instant + unsigned skippedRouted = 0; ///< hazards on buffer-ID-routed buffers: not ours + unsigned rotating = 0; ///< hazards assigned d > 1 ids (multi-buffered) + unsigned idsAssigned = 0; ///< total ids handed out (sum of d), not hazards + unsigned reused = 0; ///< assignments that took an id freed earlier in the SAME + ///< direction +}; + +/// Pre-colour the ids a macro's library implementation consumes internally. +/// Returns the number of reservations recorded. +/// +/// Runs after `buildSyncModel` and BEFORE `colorEventIds`: the colourer must see +/// these as occupied, or it hands one to a compiler hazard whose interval spans the +/// call and the library's wait is satisfied by the wrong producer. +/// +/// The span runs from the macro's first phase to its last, padded one SyncIR index +/// each side. The pad is load-bearing: without it a hazard ending exactly where the +/// call begins reads as disjoint and the collision survives. +/// +/// DELIBERATELY NOT SHARED with the oracle's own reservation walk, which re-derives +/// the same spans. Sharing would hide a bug in the span arithmetic from the gate +/// that exists to catch it. +unsigned seedHiddenMacroEvents(SyncModel &model, const SyncIRs &syncIR); + +/// Per-(src,dst) greedy first-fit interval colouring -- the classic left-edge +/// algorithm. Event flags are per-direction DISJOINT hardware, so each direction is +/// an independent interval graph. +/// +/// A multi-buffered hazard needs d ids live across its interval, so this is WEIGHTED +/// colouring and the bound is the peak SUM of demands over a point rather than the +/// peak count of intervals. That holds only because the d ids need NOT be +/// contiguous: `CreateSetWaitOpForMultiBuffer` materialises each `eventIds[i]` as an +/// independent constant selected by `slotMod == i`, and the dyn ops take the id as a +/// runtime operand that lowering passes straight through. Were contiguity required +/// this would be colouring with BANDWIDTH, which is NP-hard. +/// +/// A hazard takes EITHER all d ids OR none: fewer than d emits a selector chain +/// indexing past `eventIds`, which is an out-of-bounds read rather than a mis-sync, +/// so a short pool spills the whole hazard to a barrier. Structurally-lone barriers +/// are skipped -- they hold no id. +/// +/// Deterministic: hazards are ordered by (direction, interval start, hazard index) +/// and take the lowest free ids, except that the most recently freed id is skipped +/// while another free id exists -- an event flag is one signal deep, so alternating +/// between two ids restores a step of producer run-ahead. +/// +/// A rotating id is a RUNTIME value, recorded as -1 in the IR view, so G2/G3 cannot +/// pair a rotating set with its wait there; the syncops view is the only place those +/// ids are checked. +EventColorResult colorEventIds(SyncModel &model); + +//===----------------------------------------------------------------------===// +// Buffer routing +//===----------------------------------------------------------------------===// +// +// The order is a correctness requirement, not a convention: settle the operations, +// then route buffers, then colour what remains. Routing is per-buffer and +// ALL-OR-NOTHING -- a split buffer is a HANG, not a worse schedule. +// +// Route b iff K > 0 && b overflows its hazards' event pool && b.isWrittenBack +// +// (1) The write-back test is the per-BUFFER `Buffer::isWrittenBack`, never +// per-hazard `Hazard::hasReverseWar`. Routing is all-or-nothing, so the +// predicate must be a buffer-level fact; a per-hazard one lets two hazards on +// one buffer disagree, which is exactly the split that hangs. +// +// (2) "OVERFLOWS ITS POOL" is a PREDICTION -- colouring has not run, so there is +// no overflow to observe. For each direction this buffer's hazards +// participate in, take the peak simultaneous overlap over ALL hazards in that +// direction -- the same quantity the left-edge colourer will hit -- and +// compare against that direction's pool. +// THE TEST IS STRICT (`omega > pool`), deliberately under-predicting at the +// boundary, because the fallbacks are asymmetric. A buffer left on events +// that then overflows spills to a barrier: correct, just slower. A buffer +// routed unnecessarily consumes K, which is ONE pool shared across every pipe +// and direction and whose own exhaustion path coalesces logic ids and +// serialises co-consumers. At omega == pool the colouring fits exactly, so +// routing would be pure waste. +// +// (3) `spansLoopBoundary` IS RECORDED, NOT CONSULTED. It over-flags, because loop +// context is derived from hazard interval endpoints and a loop-carried +// hazard's interval IS its loop. Skipping buffer-ID on it would skip more +// than the evidence supports and cancel the routing the overflow test just +// asked for. +struct BufferRouteResult { + unsigned buffersConsidered = 0; + unsigned routed = 0; + unsigned hazardsCovered = 0; ///< hazards belonging to a routed buffer + unsigned skippedNoOverflow = 0; + unsigned skippedNotWrittenBack = 0; + unsigned skippedNoCapacity = 0; ///< K == 0 + /// Buffers refused because a hazard on them is not `tokenExpressible`. Refusing + /// is the whole point: routing such a buffer suppresses events no token replaces. + unsigned skippedNotExpressible = 0; + /// ALL-OR-NOTHING violations: a hazard that touches BOTH a routed and an + /// unrouted buffer, so it cannot be wholly on one mechanism. Must be 0. + unsigned splitHazards = 0; +}; + +/// Decide, per buffer, whether buffer-ID backs it. Sets `Buffer::routedToBufid` +/// and records the split check. Assigns NO ids and stamps NO mechanisms. +BufferRouteResult routeBuffers(SyncModel &model); + +} // namespace unified +} // namespace pto +} // namespace mlir + +#endif // PTO_TRANSFORMS_INSERTSYNC_UNIFIEDSYNCMODEL_H diff --git a/include/PTO/Transforms/Passes.h b/include/PTO/Transforms/Passes.h index c1d9e82a14..3339283c8c 100644 --- a/include/PTO/Transforms/Passes.h +++ b/include/PTO/Transforms/Passes.h @@ -150,10 +150,53 @@ std::unique_ptr createPTOInlineBackendHelpersPass( // Registration //===----------------------------------------------------------------------===// +/// G3: device-id legality. `bufidCapacity` is K, the number of buffer ids the arch +/// exposes: 0 on A3, 32 on A5. +/// +/// `injectFault` names a synthetic fault to append to the gate's own record list, so +/// that the gate can be shown to catch a class the IR cannot express -- an event id +/// outside [0, 8) is unrepresentable, because `pto.set_flag` spells its id as an +/// EVENT_ID0..EVENT_ID7 enum. Empty in normal use. +std::unique_ptr createPTOCheckSyncIdsPass(unsigned bufidCapacity = 0, + llvm::StringRef injectFault = ""); + +/// G2: assert no two overlapping intervals share a sync id. +std::unique_ptr +createPTOCheckSyncInterferencePass(llvm::StringRef injectFault = ""); + +/// G1-self: assert every dependence the front end reports is ordered by emitted +/// sync. `maxViolations` caps the per-function detail lines; 0 prints all. +std::unique_ptr createPTOCheckSyncSelfCoveragePass(unsigned maxViolations = 8); + +/// G1: dump the happens-before edges this run's emitted sync induces, as the +/// reference profile a later run is compared against. +std::unique_ptr createPTODumpSyncCoveragePass(); + +/// G1: assert this run establishes every ordering recorded in `referenceFile`. +std::unique_ptr createPTOCheckSyncCoveragePass(llvm::StringRef referenceFile); + +/// G4: dump this run's kernel-body sync-op counts, one line per function. +std::unique_ptr createPTODumpSyncCountsPass(); + +/// G4: assert this run's counts do not regress against `referenceFile`. +std::unique_ptr createPTOCheckSyncCountFloorPass(llvm::StringRef referenceFile); + +std::unique_ptr createPTODumpSyncExtractPass(); + +/// The unified allocator: event ids, buffer ids and barriers as three resource +/// classes over one interval-colouring problem. `bufidCapacity` is K, the number of +/// buffer ids the arch exposes -- 0 on A3, 32 on A5 -- and it is the only arch input. +std::unique_ptr +createPTOUnifiedSyncPass(unsigned bufidCapacity = 0, bool debugEnabled = false, + llvm::StringRef forceMechanism = "", + bool checkAddrReuse = false); + #undef GEN_PASS_DECL #define GEN_PASS_REGISTRATION #include "PTO/Transforms/Passes.h.inc" + + } // namespace pto } // namespace mlir diff --git a/include/PTO/Transforms/Passes.td b/include/PTO/Transforms/Passes.td index 7b08319dda..9d3f9edf86 100644 --- a/include/PTO/Transforms/Passes.td +++ b/include/PTO/Transforms/Passes.td @@ -1397,4 +1397,166 @@ def VPTOMaskSimplify let dependentDialects = ["mlir::pto::PTODialect"]; } +def PTOCheckSyncIds : Pass<"pto-check-sync-ids", "func::FuncOp"> { + let summary = "Oracle gate G3: device-id legality of the emitted synchronization"; + let description = [{ + Read-only apart from diagnostics. Checks that every static synchronization id + in the function is one the target permits: intra-core event ids inside their + per-direction pool, honouring the reserved tail, and buffer ids inside both the + hardware field and the arch's exposed buffer-id capacity, which the pass receives + from its caller and defaults to zero. Rotating ids on `set_flag_dyn` / + `wait_flag_dyn` resolve at runtime and are not checked. Fails the pass on any + violation. + }]; + + let constructor = "mlir::pto::createPTOCheckSyncIdsPass()"; +} + +def PTOCheckSyncInterference : Pass<"pto-check-sync-interference", "func::FuncOp"> { + let summary = "Oracle gate G2: no two overlapping intervals may share a sync id"; + let description = [{ + Read-only apart from diagnostics. Generalizes + `BufidSyncIdAlloc::validateNoSamePhysicalIdNesting` from buffer ids to every + id class: for each id key the live intervals must be pairwise disjoint. + Event ids are keyed by (srcPipe, dstPipe, eventId) because event flags are + per-direction disjoint hardware; buffer ids are keyed by buf_id alone + because that pool is global. Fails the pass on any nesting, on a close + without an open, or on a leaked id. + }]; + + let constructor = "mlir::pto::createPTOCheckSyncInterferencePass()"; +} + +def PTOCheckSyncSelfCoverage + : Pass<"pto-check-sync-self-coverage", "func::FuncOp"> { + let summary = "Oracle gate G1-self: every analysed dependency must be ordered by emitted sync"; + let description = [{ + ABSOLUTE, not differential -- there is no reference file. Rebuilds the + front-end SyncIR (`PTOIRTranslator`) and calls + `MemoryDependentAnalyzer::DepBetween` over the resulting def/use vectors + directly, then asserts that every dependency it reports is ordered by the + emitted sync, using the transitive closure of `computeCoverage`'s edge set + (loop-carried dependencies are checked against carried edges at d=1). + + The expected set reads none of the sync pass's decisions -- not its hazard + selection, mechanism choice, ids, or emitted ops -- so a green result is + independent of them. It is NOT a proof of correctness: pairs on mutually + exclusive `scf.if` arms are excluded; `PIPE_S`->`PIPE_S` pairs are excluded on + every arch, on compiler behaviour rather than a stated guarantee; and on A5 + `PIPE_V`->`PIPE_V` pairs are credited to the target's intra-pipe ordering + guarantee. All three exclusions are counted and reported rather than dropped. + `InsertSyncAnalysis::IsMemInfoHasDependency`, which layers policy on top of + `DepBetween`. It cannot, however, catch a dependency the front-end alias + analysis itself misses; see SyncOracleGates.h for that limit. + + Dependencies whose endpoint op is not an anchor are reported as `unmappable` + rather than dropped. + }]; + + let constructor = "mlir::pto::createPTOCheckSyncSelfCoveragePass()"; +} + +def PTODumpSyncCoverage : Pass<"pto-dump-sync-coverage", "func::FuncOp"> { + let summary = "Dump the happens-before edges the emitted sync induces (oracle gate G1, reference side)"; + let description = [{ + Read-only. Numbers every op that carries a pipe and a memory effect, then + emits the set of ordered anchor pairs the emitted synchronization + establishes, including the buffer-token ordering `rls_buf` precedes + `get_buf` for the same buffer id. + }]; + + let constructor = "mlir::pto::createPTODumpSyncCoveragePass()"; +} + +def PTOCheckSyncCoverage : Pass<"pto-check-sync-coverage", "func::FuncOp"> { + let summary = "Oracle gate G1: the new sync mode must establish every ordering InsertSync does"; + let description = [{ + Differential. Reads a reference profile produced by `pto-dump-sync-coverage` + on an InsertSync run and asserts, per function, that the current run's + happens-before edge set is a superset of the reference's. A differing anchor + signature, or a missing reference profile, is a violation. Fails the pass on + any missing ordering. + }]; + + let constructor = "mlir::pto::createPTOCheckSyncCoveragePass(\"\")"; +} + +def PTODumpSyncCounts : Pass<"pto-dump-sync-counts", "func::FuncOp"> { + let summary = "Dump kernel-body sync-op counts (oracle gate G4, reference side)"; + let description = [{ + Read-only. Emits one `[sync-count]` line per function counting the emitted + `pto.set_flag` / `pto.wait_flag` / `pto.get_buf` / `pto.rls_buf` / + `pto.barrier` ops in the function body. These are IR ops, so the emitted + `ptoas_auto_sync_tail` C++ boilerplate cannot contaminate the count. + }]; + + let constructor = "mlir::pto::createPTODumpSyncCountsPass()"; +} + +def PTOCheckSyncCountFloor : Pass<"pto-check-sync-count-floor", "func::FuncOp"> { + let summary = "Oracle gate G4: the new sync mode must not synchronize more than InsertSync"; + let description = [{ + Differential. Reads a reference profile produced by `pto-dump-sync-counts` + on an InsertSync run and asserts, per function, that the current run does not + increase (a) the total sync-op count, (b) the barrier count, or (c) the + PIPE_ALL barrier count. A missing reference profile is a violation. Fails the + pass on any regression. + }]; + + let constructor = "mlir::pto::createPTOCheckSyncCountFloorPass(\"\")"; +} + +def PTOUnifiedSync : Pass<"pto-unified-sync", "func::FuncOp"> { + let summary = "Unified intra-core sync allocator (event-ID + buffer-ID + barrier)"; + let description = [{ + Allocates intra-core synchronization for one function across all three + mechanisms from a single model, instead of committing to a mechanism before + the cost of that choice is known: event ids, drawn from a pool per + (src, dst) pipe direction; buffer-id tokens, drawn from one pool of K and + available only where K is non-zero; and PIPE_ALL barriers, the fallback that + always succeeds and orders everything. + + The dependence analysis is shared with pto-insert-sync rather than + reimplemented, so both passes see the same program and the same hazards. + Only the assignment of mechanisms and ids differs. Hazards whose direction + would overflow its event pool are routed to buffer-id tokens where the buffer + is eligible; what remains is coloured into the event pools by live interval, + and anything that still does not fit spills to a barrier. A hazard always + ends on some mechanism. + + Functions that already contain explicit synchronization are left untouched. + + Macro ops are supported. They lower to library calls that consume fixed + internal event ids which no operation in the IR names, so those (direction, id) + pairs are reserved over the call's span before any id is allocated; an id + already taken is never offered. A gate checks the result on every run, since a + collision with a library-internal id is otherwise invisible. + }]; + + let constructor = "mlir::pto::createPTOUnifiedSyncPass()"; + + let dependentDialects = [ + "mlir::pto::PTODialect", + "mlir::memref::MemRefDialect", + "mlir::arith::ArithDialect" + ]; +} + +def PTODumpSyncExtract : Pass<"pto-dump-sync-extract", "func::FuncOp"> { + let summary = "Dump the oracle's structured extraction of emitted sync ops"; + let description = [{ + Read-only. Walks the emitted `pto.set_flag` / `pto.wait_flag` / + `pto.get_buf` / `pto.rls_buf` / `pto.barrier` ops and renders one record per + line (pipes, event_id / buf_id, program order). This is the shared IR-view + surface the differential oracle gates (coverage superset, sync-count floor) + read from each compiler run. + }]; + + let constructor = "mlir::pto::createPTODumpSyncExtractPass()"; + + let dependentDialects = [ + "mlir::pto::PTODialect" + ]; +} + #endif // MLIR_DIALECT_PTO_PASSES diff --git a/lib/PTO/Transforms/BufidSync/BufidSyncAnalysis.cpp b/lib/PTO/Transforms/BufidSync/BufidSyncAnalysis.cpp index ba2209dd34..c18a617bf0 100644 --- a/lib/PTO/Transforms/BufidSync/BufidSyncAnalysis.cpp +++ b/lib/PTO/Transforms/BufidSync/BufidSyncAnalysis.cpp @@ -491,8 +491,50 @@ void BufidSyncAnalysis::insertSyncOperations() { } void BufidSyncAnalysis::optimizeSamePipeMerge() { + // DETERMINISTIC ITERATION ORDER, and it is load-bearing rather than tidiness. + // + // `op2BufSync_` is a `DenseMap`, so its iteration order + // follows the POINTER VALUES of the ops, which differ between runs of the same + // binary on the same input. The survivor below is + // `*std::min_element(ids)` over whichever ids the first-visited op happens to + // group, and it is recorded first-writer-wins, so a different visit order picks + // a different survivor and emits different code. + // + // MEASURED before this fix: `Qwen3DecodeA5/rope_kv_cache` compiled six times, + // each run pinned with `taskset -c 0`, emitted 24/24/24/23/23/23 `get_buf`. + // `mergeMap` size was stable at 4 every run while the TARGETS drifted + // (9->2/6/8, 10->3/2/6); only 3->0 and 8->3 were stable. Pinning rules out a + // thread race -- it is bucket order. This violates the determinism + // `docs/bufid_sync_a5_design.md:300` requires, breaks reproducible builds, and + // silently adds run-to-run variance to any A/B measurement taken through this + // pass. + // + // Ordering by the op's earliest bracket position (then by its smallest logic id + // as a tie-break) is stable across runs because both come from the SyncIR, not + // from the heap. The unified allocator never calls this pass, so it was already + // deterministic and is unaffected either way. + llvm::SmallVector orderedOps; + orderedOps.reserve(op2BufSync_.size()); + for (auto &entry : op2BufSync_) + orderedOps.push_back(entry.first); + auto sortKeyOf = [&](Operation *op) { + const BufSyncPipeBuild &b = op2BufSync_[op]; + int idx = std::numeric_limits::max(); + int lid = std::numeric_limits::max(); + for (const auto &list : {b.pipeBefore, b.pipeAfter}) + for (const auto &s : list) { + idx = std::min(idx, static_cast(s.syncIRIndex)); + lid = std::min(lid, s.logicId); + } + return std::make_pair(idx, lid); + }; + llvm::sort(orderedOps, [&](Operation *a, Operation *b) { + return sortKeyOf(a) < sortKeyOf(b); + }); + DenseMap> logicIdToPipeInts; - for (auto &[op, build] : op2BufSync_) { + for (Operation *op : orderedOps) { + auto &build = op2BufSync_[op]; DenseSet> seen; for (auto &s : build.pipeBefore) { auto key = std::make_pair(static_cast(s.pipe), s.logicId); @@ -508,7 +550,8 @@ void BufidSyncAnalysis::optimizeSamePipeMerge() { DenseMap mergeMap; - for (auto &[op, build] : op2BufSync_) { + for (Operation *op : orderedOps) { + auto &build = op2BufSync_[op]; DenseMap> pipeIntToLogicIds; DenseSet> seen; for (auto &s : build.pipeBefore) { diff --git a/lib/PTO/Transforms/CMakeLists.txt b/lib/PTO/Transforms/CMakeLists.txt index 75f2c25f85..06f4fd30de 100644 --- a/lib/PTO/Transforms/CMakeLists.txt +++ b/lib/PTO/Transforms/CMakeLists.txt @@ -118,6 +118,10 @@ add_mlir_dialect_library(PTOTransforms InsertSync/RemoveRedundantSync.cpp InsertSync/SyncEventIdAllocation.cpp InsertSync/SyncCodegen.cpp + InsertSync/UnifiedSyncModel.cpp + InsertSync/PTOUnifiedSync.cpp + InsertSync/SyncOracleExtract.cpp + InsertSync/SyncOracleGates.cpp BufidSync/BufidSyncPass.cpp BufidSync/BufidSyncAnalysis.cpp BufidSync/BufidSyncIdAlloc.cpp diff --git a/lib/PTO/Transforms/InsertSync/InsertSyncAnalysis.cpp b/lib/PTO/Transforms/InsertSync/InsertSyncAnalysis.cpp index 1709f10e4b..f9a9cdca2d 100644 --- a/lib/PTO/Transforms/InsertSync/InsertSyncAnalysis.cpp +++ b/lib/PTO/Transforms/InsertSync/InsertSyncAnalysis.cpp @@ -570,6 +570,20 @@ void InsertSyncAnalysis::InsertSyncOperation( SmallVector depRoots = GetMemInfoBuffers(depBaseMemInfosVec); setOp->depRootBuffers = depRoots; waitOp->depRootBuffers = depRoots; + // Keep the BaseMemInfo objects too, not just their roots. The root is the + // root ALLOCATION and is far coarser than a buffer -- many tiles share one -- + // so a consumer that needs to know *which tile* this dependency is on cannot + // recover it from `depRoots`. Both halves get the same vector, exactly as + // the roots do. + SmallVector depMemInfos; + for (const auto &memInfoPair : depBaseMemInfosVec) { + if (memInfoPair.first) + depMemInfos.push_back(memInfoPair.first); + if (memInfoPair.second) + depMemInfos.push_back(memInfoPair.second); + } + setOp->depMemInfos = depMemInfos; + waitOp->depMemInfos = depMemInfos; setOp->SetDepSyncIRIndex(frontCompound->GetIndex()); waitOp->SetDepSyncIRIndex(frontCompound->GetIndex()); diff --git a/lib/PTO/Transforms/InsertSync/InsertSyncDebug.cpp b/lib/PTO/Transforms/InsertSync/InsertSyncDebug.cpp index a01a50e1bc..663b2a8608 100644 --- a/lib/PTO/Transforms/InsertSync/InsertSyncDebug.cpp +++ b/lib/PTO/Transforms/InsertSync/InsertSyncDebug.cpp @@ -141,6 +141,14 @@ static void dumpSyncOp(llvm::raw_ostream &os, const SyncOperation *op, << getPipelineName(op->GetDstPipe()) << ">"; os << " idx=" << op->GetSyncIndex(); + // Printed only when the allocator moved this sync OFF the class its TYPE + // implies -- i.e. today, only when it routed it to BUFID. Printing the derived + // default would be redundant on every line and would churn the pinned debug + // output of every existing sync test for no information. + if (op->GetMechanism() != + SyncOperation::DefaultMechanismFor(op->GetType())) + os << " mech=" << SyncOperation::MechanismName(op->GetMechanism()); + if (op->GetForEndIndex().has_value()) os << " forEnd=" << op->GetForEndIndex().value(); diff --git a/lib/PTO/Transforms/InsertSync/MemoryDependentAnalyzer.cpp b/lib/PTO/Transforms/InsertSync/MemoryDependentAnalyzer.cpp index d2788df217..a2f927c741 100644 --- a/lib/PTO/Transforms/InsertSync/MemoryDependentAnalyzer.cpp +++ b/lib/PTO/Transforms/InsertSync/MemoryDependentAnalyzer.cpp @@ -202,6 +202,16 @@ bool MemoryDependentAnalyzer::MemAlias(const BaseMemInfo *a, } // 2. Local Memory (UB/L1) + // TWO SOURCES set these addresses, not one. PTOPlanMemory writes them back to + // allocation roots; and under --pto-level=level3 the caller writes them itself as a + // constant `addr` operand on alloc_tile, which PTOIRTranslator records with the flag + // set. "Known" means the analyzer can trust the address is real and stable, NOT that a + // particular pass produced it. + // + // Comparing these ranges is also the only way two DISTINCT roots at partially + // overlapping addresses are seen to alias, which is what makes aliasing non-transitive + // and lets a buffer clique hold more than one tile. + // // PTOPlanMemory writes physical addresses back to allocation roots. Once an // async MTE3 store has consumed the source SSA, a later allocation can reuse // the same physical range with a different alloc_tile root. Compare those diff --git a/lib/PTO/Transforms/InsertSync/PTOUnifiedSync.cpp b/lib/PTO/Transforms/InsertSync/PTOUnifiedSync.cpp new file mode 100644 index 0000000000..0f489ac1ec --- /dev/null +++ b/lib/PTO/Transforms/InsertSync/PTOUnifiedSync.cpp @@ -0,0 +1,828 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===- PTOUnifiedSync.cpp - Unified intra-core sync allocator -------------===// +// +// Allocates intra-core synchronization across all three mechanisms from one +// model, rather than choosing a mechanism up front: event ids (a pool per +// (src, dst) pipe direction), buffer-id tokens (one pool of K, exposed only where +// K > 0), and PIPE_ALL barriers as the fallback that always succeeds. +// +// The front end is REUSED, not reimplemented: Translator -> InsertSyncAnalysis -> +// MoveSyncState -> RemoveRedundantSync produce the hazard set, the same +// producer-to-consumer orderings the incumbent pass works from. Only the +// assignment of mechanisms and ids to those hazards is new. Sharing the front end +// is what makes a differential comparison against the incumbent meaningful: both +// see the same program and the same hazards. +// +// Order of work, and why it is this order: +// 1. settle every operation (loop-carried compensation, mechanism routing) +// before any id exists, since an id assigned to an op that later moves or +// disappears is worse than no id; +// 2. route buffers whose direction would overflow its event pool to buffer-id +// tokens, so the colouring below sees a smaller problem; +// 3. colour the remaining hazards' intervals into per-direction event pools, +// spilling to a barrier where the pool cannot hold them; +// 4. emit, and check the result against the oracle gates. +// +// `K` is the buffer-id capacity: 0 on A3, which disables routing entirely, and 32 +// on A5. +//===----------------------------------------------------------------------===// +#include "PTO/Transforms/Passes.h" +#include "PTO/IR/PTO.h" +#include "PTO/Transforms/InsertSync/SyncCommon.h" +#include "PTO/Transforms/InsertSync/MemoryDependentAnalyzer.h" +#include "PTO/Transforms/InsertSync/PTOIRTranslator.h" +#include "PTO/Transforms/InsertSync/InsertSyncAnalysis.h" +#include "PTO/Transforms/InsertSync/MoveSyncState.h" +#include "PTO/Transforms/InsertSync/RemoveRedundantSync.h" +#include "PTO/Transforms/InsertSync/SyncCodegen.h" +#include "PTO/Transforms/InsertSync/SyncMacroModel.h" +#include "PTO/Transforms/InsertSync/SyncOracleExtract.h" +#include "PTO/Transforms/InsertSync/SyncOracleGates.h" +#include "PTO/Transforms/InsertSync/UnifiedSyncModel.h" +#include "../BufidSync/BufidSyncAnalysis.h" +#include "../BufidSync/BufidSyncCodegen.h" +#include "../BufidSync/BufidSyncIdAlloc.h" +#include "mlir/Dialect/Func/IR/FuncOps.h" +#include "llvm/Support/raw_ostream.h" + +namespace mlir { +namespace pto { +namespace func = ::mlir::func; +#define GEN_PASS_DEF_PTOUNIFIEDSYNC +#include "PTO/Transforms/Passes.h.inc" +} // namespace pto +} // namespace mlir + +using namespace mlir; +using namespace mlir::pto; + +namespace { + +// Same gather/scatter skip condition the incumbent uses: this pass does not run +// RemoveRedundantSync on functions containing these ops. +static bool hasGatherScatterLikeOps(func::FuncOp func) { + bool found = false; + func.walk([&](Operation *op) { + if (isa(op)) { + found = true; + return WalkResult::interrupt(); + } + return WalkResult::advance(); + }); + return found; +} + +/// TEST-ONLY. Routes hazard group 0 off its default mechanism, so the BUFID value +/// is exercised end to end on kernels that would not otherwise route: set on both +/// halves of a pair, carried through the extractor, and printed by the dumper. +/// +/// Marks BOTH halves rather than one. `GetMatchSync` does propagate the mechanism +/// to the half it creates, but it built this pair inside `InsertSyncAnalysis`, +/// before any mechanism was chosen, so a later stamp on one half leaves the other +/// on its constructed default. A whole hazard group is the granularity the +/// allocator itself routes at. +/// False when `mode` names nothing, which the caller must report. Only the two +/// values below may reach the stamp: any other string used to arrive here and be +/// treated as a request to route, so a misspelling silently stamped BUFID on +/// hazard group 0 and the run then failed for an unrealised mechanism instead of +/// for the bad argument. +static bool forceMechanismOnFirstHazard(SyncOperations &ops, + llvm::StringRef mode) { + if (mode != "bufid" && mode != "bufid-spill") + return false; + if (ops.empty()) + return true; + for (auto &owned : ops.front()) { + if (!owned) + continue; + owned->SetMechanism(SyncOperation::MECHANISM::BUFID); + // "bufid-spill" additionally exercises the resource-exhaustion downgrade: + // SetPipeAll must drop the mechanism back to BARRIER along with the type, or a + // spilled barrier would keep claiming to hold a buffer-ID token it no longer + // has. + if (mode == "bufid-spill") + owned->SetPipeAll(); + } + return true; +} + +/// Detailed listing for `--check-addr-reuse-war`: every hazard the model classified +/// as sitting on a shared address, naming the two allocations that share it. +/// +/// Reads `Hazard::addressSharing` and re-derives nothing. Reports only; never fails +/// the pass -- sharing an address can be a deliberate footprint choice, and this +/// cannot see the constraints the address map was built under. +static void reportAddressReuseHazards(func::FuncOp func, + const unified::SyncModel &model) { + llvm::SmallVector lines; + std::set> addresses; + unsigned reuseCount = 0; + unsigned aliasViewCount = 0; + for (const unified::Hazard &h : model.hazards) { + if (h.addressSharing == unified::Hazard::AddressSharing::None) + continue; + bool isReuse = h.addressSharing == unified::Hazard::AddressSharing::Reuse; + if (isReuse) { + addresses.insert({static_cast(h.sharedScope), h.sharedAddress}); + ++reuseCount; + } else { + ++aliasViewCount; + } + std::string s; + llvm::raw_string_ostream os(s); + os << " !! " << (isReuse ? "reuse-ordering" : "alias-view-ordering") << " h#" + << h.index() << " scope=" << stringifyAddressSpace(h.sharedScope) + << " addr=" << h.sharedAddress << " " + << oracle::pipelineTypeName(h.srcPipe()) << "->" + << oracle::pipelineTypeName(h.dstPipe()) << "\n" + << " first alloc " << h.sharedAllocFirst->getLoc() << "\n" + << " second alloc " << h.sharedAllocSecond->getLoc() << "\n"; + lines.push_back(os.str()); + } + if (lines.empty()) + return; + oracle::emitReport([&](llvm::raw_ostream &os) { + os << "[addr-reuse-war] func=" << func.getSymName() + << " orderings_from_reuse=" << reuseCount + << " reuse_addresses=" << addresses.size() + << " alias_view_deps=" << aliasViewCount << "\n"; + for (const std::string &l : lines) + os << l; + }); +} + +struct PTOUnifiedSyncPass + : public mlir::pto::impl::PTOUnifiedSyncBase { + // Buffer-ID capacity K (0 = A3, 32 = A5). Zero disables buffer routing. + unsigned bufidCapacity = 0; + // TEST-ONLY: "" | "bufid" | "bufid-spill". See forceMechanismOnFirstHazard. + std::string forceMechanism; + /// List the hazards whose endpoints are two allocations sharing an address. + bool checkAddrReuse = false; + /// Print the allocator's model/routing/colouring reports. Off by default: the + /// reports are verbose enough to bury a caller's own output. Gate violations are + /// reported either way. + bool debugEnabled = false; + + void runOnOperation() override { + func::FuncOp func = getOperation(); + if (func.isDeclaration()) + return; + + // Do not run on functions that already carry explicit synchronization. + bool hasExplicitSync = false; + func.walk([&](Operation *op) { + if (isa(op)) { + hasExplicitSync = true; + return WalkResult::interrupt(); + } + return WalkResult::advance(); + }); + if (hasExplicitSync) + return; + + // Macro ops are supported. A macro op is one for which `getSyncMacroModel` + // returns a model; every model declares `hiddenEvents`, the fixed + // (direction, id) pairs the library's own `PtoSetWaitFlag()` calls + // use. Those uses are invisible in PTO IR -- no op names them -- so without a + // reservation the colourer would hand one to a compiler hazard whose interval + // spans the call, and nothing in the emitted IR would show it. + // + // A reservation reduces effective pool capacity, so a kernel that fit in eight + // ids can be pushed into the spill path, and at K=0 no buffer-ID routing can + // absorb that. The spill path is load-bearing for macro kernels. + + // --- Reused front-end: build the hazard set H (no id assignment) -------- + MemoryDependentAnalyzer memAnalyzer; + SyncIRs syncIR; + SyncOperations syncOpsStorage; + Buffer2MemInfoMap buffer2MemInfoMap; + + PTOIRTranslator translator(syncIR, memAnalyzer, buffer2MemInfoMap, func, + SyncAnalysisMode::NORMALSYNC); + translator.Build(); + if (syncIR.size() <= 1) + return; + + // --- Reused buffer clustering ------------------------------------------- + // One virtual buffer id per clique of aliasing tiles (buffer-keyed, not + // per-hazard). Computed on the freshly-translated SyncIR -- exactly the IR + // BufidSyncPass sees -- before any event-sync ops are inserted. Analysis + // only: no physical ids assigned, no get_buf/rls_buf emitted. + BufidSyncAnalysis bufAnalysis(syncIR, memAnalyzer, func, + /*debugEnabled=*/false); + bufAnalysis.collectDependencies(); + bufAnalysis.classifyTiles(); + bufAnalysis.allocateVirtualBufIds(); + const SmallVector &clusters = bufAnalysis.getVirtualBufIds(); + + InsertSyncAnalysis analyzer(syncIR, memAnalyzer, syncOpsStorage, func, + SyncAnalysisMode::NORMALSYNC); + analyzer.Run(/*insertBarAllAtLast=*/true); + + MoveSyncState syncMove(syncIR, syncOpsStorage); + syncMove.Run(); + + if (!hasGatherScatterLikeOps(func)) { + RemoveRedundantSync removeRedundant(syncIR, syncOpsStorage, + SyncAnalysisMode::NORMALSYNC); + removeRedundant.Run(); + } + + // Every SyncOperation carries a `mechanism` -- the resource class backing it -- + // derived from its type at construction. Only the test-only injector below + // overrides that default on a kernel the router would otherwise leave alone. + if (!forceMechanism.empty() && + !forceMechanismOnFirstHazard(syncOpsStorage, forceMechanism)) { + func.emitError() << "unknown --unified-sync-force-mechanism value '" + << forceMechanism << "'; expected bufid or bufid-spill"; + signalPassFailure(); + return; + } + + // --- The unified data model --------------------------------------------- + // One Hazard per sync group: a mechanism must be stamped on the set and the wait + // together or the pair is split across two resources and hangs. + // + // Cluster identity is `MemAlias`, the relation BufidSyncAnalysis itself uses. + // Pointer identity alone is exact but incomplete, because the tile dedup drops + // the losing BaseMemInfo objects and some hazards then retain no matching + // pointer -- hence pointer fast path, MemAlias fallback. See + // `MemInfoToClusters` for the keys this must not be simplified back to. + unified::MemInfoToClusters memInfoToClusters; + { + llvm::SmallVector hazardMemInfos; + llvm::DenseSet seen; + for (const auto &group : syncOpsStorage) + for (const auto &owned : group) + if (owned) + for (const BaseMemInfo *mi : owned->depMemInfos) + if (mi && seen.insert(mi).second) + hazardMemInfos.push_back(mi); + + for (const BaseMemInfo *mi : hazardMemInfos) { + auto &out = memInfoToClusters[mi]; + for (const VirtualBufId &vb : clusters) { + bool hit = false; + for (const TileInfo &t : vb.tiles) { + if (!t.memInfo) + continue; + if (t.memInfo == mi || + memAnalyzer.MemAlias(const_cast(mi), + const_cast(t.memInfo))) { + hit = true; + break; + } + } + if (hit && !llvm::is_contained(out, vb.logicId)) + out.push_back(vb.logicId); + } + } + } + + unified::SyncModel model = unified::buildSyncModel( + syncOpsStorage, bufidCapacity, memInfoToClusters, syncIR); + + if (checkAddrReuse) + reportAddressReuseHazards(func, model); + + // --- Settle the operations before any id is assigned -------------------- + // Synthesise the head/tail compensation pair for every loop-carried hazard + // BEFORE coloring; without it the in-body backward pair waits on a flag nothing + // raised and deadlocks on iteration 1. + // + // `fe` counts hazards flagged loop-carried by GetForEndIndex, `inv` counts + // raw-index inversion (wait before set), `pairs` the head/tail groups appended. + // fe == pairs is the invariant: one head and one tail per loop-carried hazard. + unsigned feCount = 0, invCount = 0; + for (const unified::Hazard &h : model.hazards) { + if (h.isBarrier()) + continue; + if (h.setOp()->GetForEndIndex().has_value()) + ++feCount; + if (h.waitOp() && + h.waitOp()->GetSyncIRIndex() < h.setOp()->GetSyncIRIndex()) + ++invCount; + } + // `hazards=` on the model report means "orderings the front end found". + // Capture it BEFORE compensation synthesis, which appends one group per + // loop-carried hazard -- reading it afterwards silently redefines the number. + unsigned nHazards = syncOpsStorage.size(); + + unsigned synthPairs = + unified::synthesizeLoopCompensation(model, syncOpsStorage, syncIR); + unsigned compCount = 0; + for (const unified::Hazard &h : model.hazards) + if (h.hasCompensation()) + ++compCount; + + // Decide which buffers take tokens before anything takes an event id, so the + // colourer sees the reduced demand. + unified::BufferRouteResult route = unified::routeBuffers(model); + + // REFUSED BEFORE ANY HAZARD IS TOUCHED. A split hazard names both a routed and an + // unrouted buffer, and routing is all-or-nothing, so there is no correct thing to do + // with it: stamping BUFID suppresses the event ops of the unrouted half without + // giving it a token, and declining leaves the routed half's buffer half-converted. + // The plan is rejected whole instead, while `model.hazards` is still untouched -- + // `RouteToBufid` below is the first mutation, and it must not run on a plan this + // shape. + // + // Closing routing over each buffer-hazard component would let the plan proceed with + // a smaller routed set. That is a strictly better answer and a separate change; it is + // not needed for correctness, and refusing is what makes the invariant hold today. + if (route.splitHazards != 0) { + func.emitError() + << "unified allocator: " << route.splitHazards + << " hazard(s) span both a routed and an unrouted buffer, and routing is " + "all-or-nothing, so no mechanism can carry them wholly -- refusing the " + "routing plan rather than emitting a partially converted one"; + return signalPassFailure(); + } + + // --- Buffer-ID emission --------------------------------------------------- + // + // Three things must land TOGETHER; any two without the third is the + // split-mechanism shape that hangs: + // 1. stamp BUFID on every hazard of a routed buffer, and suppress its event + // ops -- codegen dispatches on TYPE, so stamping alone changes nothing; + // 2. the colourer skips them, so they take no event id; + // 3. get_buf/rls_buf are emitted for them. + // (1) and (2) are here; (3) is after the event codegen below. + unsigned routedHazards = 0; + llvm::SmallDenseSet routedLogicIds; + for (const unified::Buffer &b : model.buffers) + if (b.routedToBufid) + routedLogicIds.insert(b.logicId); + if (!routedLogicIds.empty()) { + for (unified::Hazard &h : model.hazards) { + if (h.isBarrier() || h.bufferClusters.empty()) + continue; + bool onRouted = llvm::any_of(h.bufferClusters, [&](int c) { + return routedLogicIds.contains(c); + }); + if (!onRouted) + continue; + h.RouteToBufid(); + ++routedHazards; + } + } + + // Pre-colour the ids a macro's library implementation consumes internally, + // before the greedy scan can hand one out. + unsigned hiddenReserved = unified::seedHiddenMacroEvents(model, syncIR); + unified::EventColorResult alloc = unified::colorEventIds(model); + + auto syncOpRecords = oracle::extractFromSyncOps(syncOpsStorage); + + oracle::DeviceIdLimits limits; + limits.bufIdCapacity = bufidCapacity; + auto idViolations = oracle::checkDeviceIdLegality(syncOpRecords, limits); + // A macro's library implementation consumes event ids that no PTO op names, so + // an id handed to a hazard spanning the call collides invisibly. `colorEventIds` + // already refuses a reserved id; this re-checks the result independently, because + // no other gate can see a use inside a library call. Folded into the same list as + // G3 because it is the same question -- is this id legal here -- answered against + // a different source. + for (const auto &v : + oracle::checkMacroHiddenEventCollisions(syncIR, syncOpRecords)) + idViolations.push_back(v); + auto interference = oracle::checkNonInterference(syncOpRecords); + + // Gate violations are reported whether or not debug is on: they precede a + // signalPassFailure, so suppressing them would leave a failing pass with no reason. + // Nothing is printed when they are empty. + if (!idViolations.empty() || oracle::countErrors(interference) > 0) + oracle::emitReport([&](llvm::raw_ostream &os) { + oracle::printIdViolations(os, func.getSymName(), "syncops", + syncOpRecords.size(), idViolations); + oracle::printInterferenceViolations(os, func.getSymName(), "syncops", + syncOpRecords.size(), interference); + }); + + unsigned nShareReuse = 0; + unsigned nShareAlias = 0; + for (const unified::Hazard &h : model.hazards) { + if (h.addressSharing == unified::Hazard::AddressSharing::Reuse) + ++nShareReuse; + else if (h.addressSharing == unified::Hazard::AddressSharing::AliasView) + ++nShareAlias; + } + + // One atomic report: nested func passes run in parallel (see emitReport). + if (debugEnabled) + oracle::emitReport([&](llvm::raw_ostream &os) { + os << "[unified-sync model] kernel=" << func.getSymName() + << " K=" << bufidCapacity << " hazards=" << nHazards + << " share_reuse=" << nShareReuse + << " share_alias=" << nShareAlias << "\n"; + oracle::printSyncOpRecords(os, func.getSymName(), syncOpRecords); + oracle::printIdViolations(os, func.getSymName(), "syncops", + syncOpRecords.size(), idViolations); + oracle::printInterferenceViolations(os, func.getSymName(), "syncops", + syncOpRecords.size(), interference); + os << "[unified-sync reachable] buffer clusters (virtual buf ids) = " + << clusters.size() << "\n"; + unsigned shownC = 0; + for (const VirtualBufId &vb : clusters) { + os << " cluster#" << vb.logicId + << " scope=" << stringifyAddressSpace(vb.scope) + << " aliasing_tiles=" << vb.tiles.size() << "\n"; + if (++shownC >= 6) + break; + } + os << "[unified-sync reachable] bufid emission ops reachable: " + << pto::GetBufOp::getOperationName() << ", " + << pto::RlsBufOp::getOperationName() << "\n"; + unified::printSyncModel(os, func.getSymName(), model); + os << "[unified-sync routing] func=" << func.getSymName() + << " buffer routing: considered=" + << route.buffersConsidered << " routed=" << route.routed + << " hazards_covered=" << route.hazardsCovered + << " skipped(no_overflow=" << route.skippedNoOverflow + << " not_written_back=" << route.skippedNotWrittenBack + << " K=0:" << route.skippedNoCapacity << ")" + << " not_expressible=" << route.skippedNotExpressible + << " split=" << route.splitHazards + << (route.splitHazards == 0 ? " OK" : " !! SPLIT-MECHANISM") << "\n"; + for (const unified::Buffer &b : model.buffers) + if (b.routedToBufid) + os << " ROUTE " << func.getSymName() << " b#" << b.logicId + << " -> bufid" + << " omega=" << b.predictedPeakOverlap << ">pool=" + << b.smallestPool << " wb=yes lbufid=[" << b.bufidLoop.start << "," + << b.bufidLoop.end << "]\n"; + // A divergence between `fe` and `inv` means compensation was synthesised for + // a FORWARD hazard, as `prefetch_disjoint_slots` shows. + // + // WARNING, not error, and the incumbent is the reason: it primes forward + // hazards too, so a divergence is not by itself wrong, and erroring would + // refuse multi-tile kernels that `--enable-insert-sync` compiles. + if (feCount != invCount) + os << "[unified-sync compensation] !! predicate-divergence func=" + << func.getSymName() << " fe=" << feCount << " inv=" << invCount + << " : compensation synthesised for " << (feCount - invCount) + << " hazard(s) whose in-body set precedes its wait (forward, not " + "index-inverted). Not an error -- the incumbent primes these too -- but " + "the two predicates are not interchangeable.\n"; + os << "[unified-sync compensation] loop-compensation: fe=" << feCount + << " inv=" << invCount << " pairs=" << synthPairs + << " heads=" << synthPairs << " tails=" << synthPairs + << " linked=" << compCount + << (feCount == synthPairs && synthPairs == compCount ? " OK" : " MISMATCH") + << "\n"; + os << "[unified-sync coloring] interval coloring: assigned=" << alloc.assigned + << " overflow=" << alloc.overflow + << " spilled_to_barrier=" << alloc.spilledToBarrier + << " hidden_reserved=" << hiddenReserved + << " reused=" << alloc.reused + << " skipped_routed=" << alloc.skippedRouted + << " rotating=" << alloc.rotating + << " ids_assigned=" << alloc.idsAssigned + << " (per-direction first-fit; ids reused across disjoint intervals)\n"; + + // --- Placement of the synthesised compensation pairs ------------------ + // SyncCodegen walks syncIR's pipe lists, so the pair must BE in them or it + // is dropped silently at emission. The load-bearing part is whether the tail + // PRECEDES any existing loop-end SET there; a tail after a set is the + // broken-sequence case, reported ORDER_BAD. + unsigned placedHead = 0, placedTail = 0, orderOk = 0, orderBad = 0; + os << "[unified-sync placement] placement into syncIR pipe lists:\n"; + for (const unified::Hazard &h : model.hazards) { + if (!h.hasCompensation()) + continue; + const SyncOperation *head = h.headOp(); + const SyncOperation *tail = h.tailOp(); + unsigned b = head->GetSyncIRIndex(); + unsigned e = tail->GetSyncIRIndex(); + const auto &before = syncIR[b]->pipeBefore; + const auto &after = syncIR[e]->pipeAfter; + int hPos = -1, tPos = -1, firstSet = -1; + for (size_t i = 0; i < before.size(); ++i) + if (before[i] == head) { + hPos = static_cast(i); + break; + } + for (size_t i = 0; i < after.size(); ++i) { + if (after[i] == tail && tPos < 0) + tPos = static_cast(i); + if (firstSet < 0 && after[i] != tail && + after[i]->GetType() == SyncOperation::TYPE::SET_EVENT) + firstSet = static_cast(i); + } + if (hPos >= 0) + ++placedHead; + if (tPos >= 0) + ++placedTail; + bool ok = (tPos >= 0) && (firstSet < 0 || tPos < firstSet); + ok ? ++orderOk : ++orderBad; + os << " pair h#" << h.index() + << " id=" << (h.hasEventId() ? h.setOp()->eventIds.front() : -1) + << " head@syncIR[" << b << "].pipeBefore[" << hPos << "/" + << before.size() << "] tail@syncIR[" << e << "].pipeAfter[" << tPos + << "/" << after.size() << "] firstLoopEndSet=" << firstSet + << (ok ? " ORDER_OK" : " ORDER_BAD(tail after set)") << "\n"; + } + os << "[unified-sync placement] placed: heads=" << placedHead + << " tails=" << placedTail << " of pairs=" << synthPairs + << " order_ok=" << orderOk << " order_bad=" << orderBad + << ((placedHead == synthPairs && placedTail == synthPairs && + orderBad == 0) + ? " OK" + : " MISMATCH") + << "\n"; + + os << "[unified-sync codegen] codegen WIRED: emitting set_flag/wait_flag/" + "barrier from the colored model\n"; + }); + + if (!idViolations.empty() || !interference.empty()) { + func.emitError() << "unified allocator failed an oracle gate: " + << idViolations.size() << " illegal id(s), " + << interference.size() << " interfering id(s)"; + return signalPassFailure(); + } + + // --- Emission ----------------------------------------------------------- + // Driven exactly as the incumbent drives it (PTOInsertSync.cpp): construct + // over the same `syncIR` and Run(). Codegen walks syncIR's pipeBefore/ + // pipeAfter lists -- which is why the step above had to place the synthesised + // head/tail pairs there, or they would be dropped silently here. + // + // Runs AFTER the in-pass gates: an allocation the oracle rejects is not + // emitted. + SyncCodegen codegen(syncIR, func, SyncAnalysisMode::NORMALSYNC); + codegen.Run(); + // Fail the pass on an unrealised hazard. Only the unified path signals failure: + // insert-sync provably cannot reach the guard, so its shipping behaviour is + // untouched (the diagnostic itself is emitted from SyncCodegen either way). + if (codegen.sawUnrealisedHazard()) + signalPassFailure(); + + // --- (3) Emit get_buf/rls_buf for the routed buffers --------------------- + // + // REUSE, NOT REBUILD. `BufidSyncAnalysis::insertSyncOperations` + + // `BufidSyncIdAlloc` + `BufidSyncCodegen` are the production pipeline, and it + // already runs over the same SyncIR and the same clusters routing ran on. The + // only addition is a FILTER: BufidSync emits for every cluster, only the routed + // ones are wanted, and `BufSyncOperation` carries `logicId`. + // + // PHYSICAL IDS ARE DELEGATED to `BufidSyncIdAlloc`, not re-colored here. It is + // the same greedy left-edge scan but over ONE pool of K shared across all + // directions, unlike `colorEventIds` which is per direction. It also owns the + // exhaustion path (`needsReuse` -> `reuseIds` -> `compactPhysicalIds`, which + // coalesces logic ids rather than failing) and + // `validateNoSamePhysicalIdNesting`, and its interval model is + // `computeLifeIntervals`' union span -- the same semantics the event colourer is + // aligned to. Duplicating it would fork them. + if (routedHazards > 0) { + // --- Token sites come from the ALLOCATOR'S HAZARDS, not from + // --- BufidSyncAnalysis's own dependence set ------------------------------ + // + // `BufidSyncAnalysis::collectDependencies` enumerates FORWARD pairs only + // (`for j = i + 1`), so a loop-carried hazard -- whose producer sits after its + // consumer in program order and whose ordering crosses the back edge -- can + // never appear in `depPairs_`. Feeding the emitter from that set while routing + // from this one suppressed such a hazard's event ops and emitted no token for + // it, losing the ordering with nothing to report it. + // + // A token DOES carry a loop-carried ordering, and needs no priming pair to do + // it: the counter starts free, so iteration N+1's `get_buf` blocks on + // iteration N's `rls_buf`. Only the emission was missing. + // + // So the sites are derived here, from each routed hazard, and BOTH endpoints + // are bracketed -- four anchors where the producer and consumer are distinct + // ops. Where several ops on the right pipe alias the cluster, ALL of them are + // bracketed rather than one being chosen: over-bracketing spends a token, + // under-bracketing loses an ordering, and those are not symmetric. + auto &op2BufSync = bufAnalysis.getOp2BufSync(); + op2BufSync.clear(); + + // Cluster -> its tiles' memInfos, for aliasing tests below. + llvm::DenseMap> clusterMemInfos; + for (const VirtualBufId &vb : clusters) + if (routedLogicIds.contains(vb.logicId)) + for (const TileInfo &t : vb.tiles) + if (t.memInfo) + clusterMemInfos[vb.logicId].push_back(t.memInfo); + + auto aliasesCluster = [&](const CompoundInstanceElement *cp, int logicId) { + auto it = clusterMemInfos.find(logicId); + if (it == clusterMemInfos.end()) + return false; + for (const auto *vec : {&cp->defVec, &cp->useVec}) + for (const BaseMemInfo *mi : *vec) { + if (!mi) + continue; + for (const BaseMemInfo *tm : it->second) + if (mi == tm || + memAnalyzer.MemAlias(const_cast(mi), + const_cast(tm))) + return true; + } + return false; + }; + + // Bracket one op for one (pipe, cluster), de-duplicated: the emitter would + // otherwise emit the same get/rls twice for an op touching the cluster twice. + unsigned keptGet = 0, keptRls = 0; + auto bracket = [&](Operation *op, PipelineType pipe, int logicId, + unsigned irIdx) { + if (!op) + return; + auto &build = op2BufSync[op]; + auto present = [&](const llvm::SmallVector &v, + BufSyncType ty) { + for (const BufSyncOperation &s : v) + if (s.type == ty && s.logicId == logicId && s.pipe == pipe) + return true; + return false; + }; + if (!present(build.pipeBefore, BufSyncType::GET_BUF)) { + build.pipeBefore.push_back( + {BufSyncType::GET_BUF, pipe, logicId, irIdx, irIdx}); + ++keptGet; + } + if (!present(build.pipeAfter, BufSyncType::RLS_BUF)) { + build.pipeAfter.push_back( + {BufSyncType::RLS_BUF, pipe, logicId, irIdx, irIdx}); + ++keptRls; + } + }; + + unsigned bracketedHazards = 0, unbracketedHazards = 0; + for (const unified::Hazard &h : model.hazards) { + if (h.mechanism() != SyncOperation::MECHANISM::BUFID || h.isBarrier()) + continue; + bool anySite = false; + for (int c : h.bufferClusters) { + if (!routedLogicIds.contains(c)) + continue; + // Both ends: producer sites on srcPipe, consumer sites on dstPipe. + for (const auto &element : syncIR) { + auto *cp = dyn_cast_or_null(element.get()); + if (!cp || !cp->elementOp || !aliasesCluster(cp, c)) + continue; + if (cp->kPipeValue == h.srcPipe() || cp->kPipeValue == h.dstPipe()) { + bracket(cp->elementOp, cp->kPipeValue, c, cp->GetIndex()); + anySite = true; + } + } + } + anySite ? ++bracketedHazards : ++unbracketedHazards; + } + + // Checked BEFORE the emptiness guard below, deliberately. If bracketing found no + // site for ANY routed cluster then `op2BufSync` is empty, the whole block below is + // skipped -- and `Hazard::RouteToBufid` has already suppressed those hazards' event + // ops. That is the total-failure form of exactly what this counts, so testing it + // inside the block would miss the worst case. + if (unbracketedHazards != 0) { + func.emitError() << "unified allocator: " << unbracketedHazards + << " routed hazard(s) got no token bracket while their event ops " + "were suppressed, so those orderings are carried by neither " + "mechanism"; + return signalPassFailure(); + } + + if (!op2BufSync.empty()) { + BufidSyncIdAlloc idAlloc(bufAnalysis.getVirtualBufIds(), op2BufSync, + syncIR, bufidCapacity, /*debugEnabled=*/false); + idAlloc.computeLifeIntervals(); + idAlloc.linearScanAllocate(); + idAlloc.compactPhysicalIds(); + if (idAlloc.needsReuse()) { + idAlloc.reuseIds(); + idAlloc.compactPhysicalIds(); + } + + // RE-CHECKED, because `reuseIds` is not guaranteed to converge: it leaves its + // `while (maxPhysicalIdUsed_ >= physicalBufIdCount_)` loop with the condition + // still true when no signature group holds two logic ids to merge. + // `BufidSyncCodegen` then casts the + // physical id to uint32_t with no range test, and an id past the pool aliases a + // different buffer's token -- corrupting an ordering that has nothing to do with + // this one. No oracle gate catches it either: the buf-id range check lives in the + // IRSyncRecord overload of `checkDeviceIdLegality`, which only PTOCheckSyncIds + // runs, while this pass calls the SyncOpRecord overload that has no buf-id branch. + if (idAlloc.needsReuse()) { + func.emitError() + << "unified allocator: buffer-id demand exceeds the pool of " + << bufidCapacity + << " and id reuse did not resolve it, so a physical id would fall outside " + "the pool and alias another buffer's token"; + return signalPassFailure(); + } + + bufAnalysis.setLogicToPhysicalId(idAlloc.getLogicToPhysical()); + bufAnalysis.mergeGetRls(); + + // VALIDATED AFTER THE MERGE, which is the order the production pass uses and the + // opposite of what this did before. `mergeGetRls` cancels an rls/get pair on one + // (logicId, pipe), turning two point-holds of a physical id into a single hold + // spanning both sites -- precisely the property being validated. Validating first + // inspects a structure that is not the one emitted. The validator reads + // `op2BufSync_` by reference, so after the merge it sees what codegen will see. + // + // A violation is a get_buf on an id already held, waiting for a release issued + // later in program order. That hangs, so it fails the pass rather than being + // reported. + std::string nestErr; + const bool nestOk = idAlloc.validateNoSamePhysicalIdNesting(&nestErr); + if (!nestOk) { + func.emitError() << "unified allocator: buffer-id token nesting is invalid: " + << nestErr; + return signalPassFailure(); + } + + BufidSyncCodegen bufCodegen(func, op2BufSync, idAlloc); + const bool emitOk = succeeded(bufCodegen.run()); + if (!emitOk) { + func.emitError() << "unified allocator: buffer-id emission failed, so the " + "routed hazards have neither events nor tokens"; + return signalPassFailure(); + } + + // ALL-OR-NOTHING. Every event op belonging to a routed hazard must have been + // suppressed, or codegen emitted a set/wait for a buffer whose ordering is + // carried by the token -- the counter cycle missing a step, i.e. a hang. Counted + // here and failed below, rather than only printed under the debug flag. + // Keyed on the ROUTED set, not on mechanism alone: `forceMechanismOnFirstHazard` + // stamps BUFID via SetMechanism without calling RouteToBufid, so those ops keep + // their event ops live by design and their cluster never enters `routedLogicIds`. + // Counting them here would fail a forced run with a message blaming routing. + unsigned suppressed = 0, leaked = 0; + for (const unified::Hazard &h : model.hazards) { + if (h.mechanism() != SyncOperation::MECHANISM::BUFID) + continue; + if (!llvm::any_of(h.bufferClusters, [&](int c) { + return routedLogicIds.contains(c); + })) + continue; + for (SyncOperation *op : + {h.setOp(), h.waitOp(), h.headOp(), h.tailOp()}) { + if (!op) + continue; + op->uselessSync ? ++suppressed : ++leaked; + } + } + + // Failed, not merely reported. A leaked event op means a routed hazard kept a + // set/wait alongside its token: the counter cycle missing a step, i.e. a hang. + // Previously computed and printed under the debug flag only, so a regression + // produced a warning string in a report nobody reads and a successful compile. + if (leaked != 0) { + func.emitError() << "unified allocator: " << leaked + << " event op(s) of a routed hazard were not suppressed, so " + "that ordering is carried by both mechanisms at once"; + return signalPassFailure(); + } + + if (debugEnabled) + oracle::emitReport([&](llvm::raw_ostream &os) { + os << "[unified-sync bufid] func=" << func.getSymName() + << " bufid emission: routed_buffers=" << routedLogicIds.size() + << " routed_hazards=" << routedHazards + << " anchor_ops=" << op2BufSync.size() << " get=" << keptGet + << " rls=" << keptRls + << " physical_ids=" << idAlloc.getLogicToPhysical().size() + // These read OK whenever the report prints, because a violation + // returned above. Written as tests anyway so the line keeps saying + // what it measures rather than asserting it. + << " nesting=" << (nestOk ? "OK" : "VIOLATION") + << " emit=" << (emitOk ? "OK" : "FAILED") + << " bracketed_hazards=" << bracketedHazards + << " unbracketed_hazards=" << unbracketedHazards + << (unbracketedHazards == 0 ? "" : " !! UNBRACKETED") + << " suppressed_event_ops=" << suppressed + << " leaked_event_ops=" << leaked + << (leaked == 0 ? " ALL-OR-NOTHING-OK" : " !! SPLIT-MECHANISM") + << "\n"; + }); + } + } + } +}; + +} // namespace + +std::unique_ptr +mlir::pto::createPTOUnifiedSyncPass(unsigned bufidCapacity, bool debugEnabled, + llvm::StringRef forceMechanism, + bool checkAddrReuse) { + auto pass = std::make_unique(); + pass->bufidCapacity = bufidCapacity; + pass->debugEnabled = debugEnabled; + pass->forceMechanism = forceMechanism.str(); + pass->checkAddrReuse = checkAddrReuse; + return pass; +} diff --git a/lib/PTO/Transforms/InsertSync/SyncCodegen.cpp b/lib/PTO/Transforms/InsertSync/SyncCodegen.cpp index 8ef06d6c8f..5272fa2106 100644 --- a/lib/PTO/Transforms/InsertSync/SyncCodegen.cpp +++ b/lib/PTO/Transforms/InsertSync/SyncCodegen.cpp @@ -272,6 +272,47 @@ void SyncCodegen::SyncInsert(IRRewriter &rewriter, Operation *op, if (sync->GetType() == SyncOperation::TYPE::PIPE_BARRIER) { CreateBarrierOp(rewriter, insertAnchorOp, sync, forceBefore); } else if (sync->isSyncSetType() || sync->isSyncWaitType()) { + // A set/wait hazard with NO event id has no mechanism realised: nothing below + // will emit anything for it, and the hazard's ordering simply vanishes from the + // kernel. That must be LOUD. + // + // It used to be silent, and it was reachable two ways. (1) The multi-id fallback + // iterates `eventIds` -- zero ids means zero flags emitted, then `return`. + // (2) `--unified-sync-force-mechanism=bufid` stamps MECHANISM::BUFID, which makes + // `colorEventIds` skip the hazard, while nothing routes a buffer for it -- so the + // pair arrived here empty and disappeared, with ptoas exiting 0 and every gate + // reporting clean. Those are one defect, not two. + // + // "No id" is never a resting state: a hazard must end on SOME mechanism -- an + // event id, a buffer token, or a barrier -- which is exactly what + // `Hazard::SpillToBarrier` exists to guarantee (it converts the hazard AND + // suppresses its synthesised compensation, because `SetEventId` stamps all four + // ops alike). This guard is what makes that invariant checked rather than assumed: + // the synthesised compensation lives in `pipeBefore`/`pipeAfter`, the two containers the + // incumbent's own degradation ladder does not walk, so nothing else would catch a + // future path that left one live and unstamped. + // + // Reachability on the shipping path was traced and adversarially reviewed: + // `--enable-insert-sync` cannot arrive here empty, because the backward-match + // synthetics are always stamped with an id before being published into the sync + // lists. So this is hardening for the unified path, not a fix to insert-sync. + // + // DELIBERATELY NO ASSERT. An assertion ahead of the diagnostic aborts an + // assertions build at precisely the point this guard exists to report, so the + // negative test that pins the message sees a crash instead. Failure propagates + // through `sawUnrealisedHazard_`, which the unified pass turns into + // signalPassFailure, and that works identically with assertions on or off. + if (sync->eventIds.empty()) { + sawUnrealisedHazard_ = true; + func_.emitError() + << "sync codegen: a " << SyncOperation::TypeName(sync->GetType()) + << " hazard (src pipe " << int(sync->GetActualSrcPipe()) << " -> dst pipe " + << int(sync->GetActualDstPipe()) << ", syncIndex " + << sync->GetSyncIndex() << ") reached emission with no event id, so no mechanism was realised for it " + "and its ordering would be silently dropped. A hazard must end on an " + "event id, a buffer token, or a barrier."; + return; + } if (sync->eventIds.size() == 1) { CreateSetWaitOpForSingleBuffer(rewriter, insertAnchorOp, sync, forceBefore); } else { @@ -361,8 +402,26 @@ void SyncCodegen::CreateSetWaitOpForMultiBuffer(IRRewriter &rewriter, // fall back to a static set/wait on the first event id. This preserves // correctness at the cost of forgoing per-slot pipelining. if (!sync->slotSSAExpr || sync->eventIds.empty()) { - auto eventId = getEventAttr(rewriter, sync->eventIds[0]); - createSetOrWaitFlagOp(rewriter, op, sync, srcPipe, dstPipe, eventId); + // A MULTI-ID OP WITH NO SLOT SSA MUST EMIT ONE FLAG PER ID, NOT ONE FLAG. + // This is the loop-carried COMPENSATION of a rotating hazard: the head/tail + // are synthesised, so they have no slot expression, but the in-body pair they + // prime is `wait_flag_dyn` selecting eventIds[slot % N]. Priming only + // eventIds[0] leaves every other slot's flag unset, and the first iteration + // that lands on slot != 0 BLOCKS FOREVER -- the iteration-1 deadlock the carried + // exists to catch, reintroduced through the rotation path. + // + // The incumbent reaches the same emitted shape by a different route: its + // `UpdateBackwardMatchSync(setFlag, waitFlag, eventId)` takes ONE id and + // builds one compensation pair PER id, so each of its head ops carries a + // single id and never lands here. Ours carries the whole id list on one pair + // (SetEventIds stamps all four ops alike), so the fan-out happens here. + // Either way the emitted IR is d static flags -- verified identical. + // + // Ops with exactly one id are unaffected. + for (int id : sync->eventIds) { + auto eventId = getEventAttr(rewriter, id); + createSetOrWaitFlagOp(rewriter, op, sync, srcPipe, dstPipe, eventId); + } return; } diff --git a/lib/PTO/Transforms/InsertSync/SyncCommon.cpp b/lib/PTO/Transforms/InsertSync/SyncCommon.cpp index e1efcf7d65..7d5efbfe99 100644 --- a/lib/PTO/Transforms/InsertSync/SyncCommon.cpp +++ b/lib/PTO/Transforms/InsertSync/SyncCommon.cpp @@ -89,6 +89,40 @@ std::string SyncOperation::TypeName(SyncOperation::TYPE t) { return ""; } +SyncOperation::MECHANISM +SyncOperation::DefaultMechanismFor(SyncOperation::TYPE t) { + switch (t) { + case TYPE::SET_EVENT: + case TYPE::WAIT_EVENT: + return MECHANISM::EVENT; + case TYPE::PIPE_BARRIER: + case TYPE::PIPE_BARRIER_CUBE: + case TYPE::PIPE_BARRIER_VECTOR: + return MECHANISM::BARRIER; + case TYPE::SYNC_BLOCK_SET: + case TYPE::SYNC_BLOCK_WAIT: + case TYPE::SYNC_BLOCK_ALL: + return MECHANISM::BLOCK; + } + llvm_unreachable("Not supported sync type"); + return MECHANISM::EVENT; +} + +std::string SyncOperation::MechanismName(SyncOperation::MECHANISM m) { + switch (m) { + case MECHANISM::EVENT: + return "event"; + case MECHANISM::BUFID: + return "bufid"; + case MECHANISM::BARRIER: + return "barrier"; + case MECHANISM::BLOCK: + return "block"; + } + llvm_unreachable("Not supported sync mechanism"); + return ""; +} + std::string SyncOperation::GetCoreTypeName(TCoreType t) const { static std::map coreTypeNameMap = { {TCoreType::CUBE, "CUBE"}, @@ -122,10 +156,17 @@ SyncOperation::GetMatchSync(unsigned index) const { auto res = std::make_unique(newType, this->srcPipe_, this->dstPipe_, kSyncIndex_, index, this->forEndIndex_); + // The matched half must sit in the SAME resource class -- a BUFID set with an + // EVENT wait is not a pair, it is two unrelated syncs. The constructor derived + // a default from `newType`, which is right for EVENT/BARRIER/BLOCK and WRONG + // for BUFID (not derivable from TYPE), so it must be propagated explicitly. + res->mechanism_ = this->mechanism_; res->eventIds = this->eventIds; res->depRootBuffers = this->depRootBuffers; + res->depMemInfos = this->depMemInfos; res->eventIdNum = this->eventIdNum; res->isCompensation = this->isCompensation; + res->compensationOf = this->compensationOf; res->autoSyncTailBarrier = this->autoSyncTailBarrier; res->SetDepSyncIRIndex(this->GetDepSyncIRIndex()); // Slot info: propagate as a default; callers that know the matched side's @@ -139,6 +180,11 @@ SyncOperation::GetMatchSync(unsigned index) const { void SyncOperation::SetPipeAll() { // set current sync to pipe_all this->type_ = TYPE::PIPE_BARRIER; + // This is the resource-exhaustion downgrade, so the mechanism must fall with + // the type. Leaving it would let a spilled barrier keep reporting EVENT (or, + // once the allocator routes them, BUFID) and hand the codegen a sync it would + // try to back with an id that is no longer held. + this->mechanism_ = DefaultMechanismFor(this->type_); // [修正] 使用 pto::PipelineType this->srcPipe_ = PipelineType::PIPE_ALL; this->dstPipe_ = PipelineType::PIPE_ALL; diff --git a/lib/PTO/Transforms/InsertSync/SyncEventIdAllocation.cpp b/lib/PTO/Transforms/InsertSync/SyncEventIdAllocation.cpp index 98e0457993..b1f85325d1 100644 --- a/lib/PTO/Transforms/InsertSync/SyncEventIdAllocation.cpp +++ b/lib/PTO/Transforms/InsertSync/SyncEventIdAllocation.cpp @@ -133,6 +133,13 @@ size_t SyncEventIdAllocation::GetCompilerAvailableEventIdNum(const SyncOperation return kTotalEventIdNum; } +uint64_t SyncEventIdAllocation::GetReservedEventIdNum(PipelineType src, + PipelineType dst) { + auto it = reservedEventIdNum.find({src, dst}); + return it == reservedEventIdNum.end() ? 0 : it->second; +} + + void SyncEventIdAllocation::SetEventId(SyncOperation *sync) { const size_t poolSize = getEventIdPoolSize(sync, reservedBlockSyncEventIdNum); const size_t availableEventIdNum = GetCompilerAvailableEventIdNum(sync); diff --git a/lib/PTO/Transforms/InsertSync/SyncOracleExtract.cpp b/lib/PTO/Transforms/InsertSync/SyncOracleExtract.cpp new file mode 100644 index 0000000000..df3ee83485 --- /dev/null +++ b/lib/PTO/Transforms/InsertSync/SyncOracleExtract.cpp @@ -0,0 +1,284 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===- SyncOracleExtract.cpp - Shared sync-op extractor (oracle) ----------===// +// +// Implements the two extraction views (see SyncOracleExtract.h) that the gates +// read a compilation's emitted sync through. +//===----------------------------------------------------------------------===// + +#include "PTO/Transforms/InsertSync/SyncOracleExtract.h" +#define GEN_PASS_DEF_PTODUMPSYNCEXTRACT +#include "PTO/Transforms/Passes.h" +#include "PTO/IR/PTO.h" +#include "PTO/IR/PTOSyncUtils.h" +#include "llvm/Support/Mutex.h" + +namespace mlir { +namespace pto { +namespace func = ::mlir::func; + +#include "PTO/Transforms/Passes.h.inc" + +} // namespace pto +} // namespace mlir + +using namespace mlir; +using namespace mlir::pto; + +void mlir::pto::oracle::emitReport( + llvm::function_ref body) { + std::string buffer; + { + llvm::raw_string_ostream os(buffer); + body(os); + } + // stderr, NOT stdout: stdout carries ptoas's product (the emitted CCEC C++). + // Reports written there are spliced into the kernel source, which then + // stops compiling. + static llvm::sys::SmartMutex mutex; + llvm::sys::SmartScopedLock lock(mutex); + llvm::errs() << buffer; + llvm::errs().flush(); +} + +llvm::StringRef mlir::pto::oracle::pipelineTypeName(PipelineType pipe) { + switch (pipe) { + case PipelineType::PIPE_S: + return "S"; + case PipelineType::PIPE_V: + return "V"; + case PipelineType::PIPE_M: + return "M"; + case PipelineType::PIPE_MTE1: + return "MTE1"; + case PipelineType::PIPE_MTE2: + return "MTE2"; + case PipelineType::PIPE_MTE3: + return "MTE3"; + case PipelineType::PIPE_ALL: + return "ALL"; + default: + return "?"; + } +} + +/// IR pipe enum -> the InsertSync PipelineType the gates reason about. Written +/// as an explicit switch rather than a numeric cast: the two enums happen to +/// agree today, and nothing enforces that they keep agreeing. +static PipelineType toPipelineType(pto::PIPE pipe) { + switch (pipe) { + case pto::PIPE::PIPE_S: + return PipelineType::PIPE_S; + case pto::PIPE::PIPE_V: + return PipelineType::PIPE_V; + case pto::PIPE::PIPE_M: + return PipelineType::PIPE_M; + case pto::PIPE::PIPE_MTE1: + return PipelineType::PIPE_MTE1; + case pto::PIPE::PIPE_MTE2: + return PipelineType::PIPE_MTE2; + case pto::PIPE::PIPE_MTE3: + return PipelineType::PIPE_MTE3; + case pto::PIPE::PIPE_ALL: + return PipelineType::PIPE_ALL; + case pto::PIPE::PIPE_MTE4: + return PipelineType::PIPE_MTE4; + case pto::PIPE::PIPE_MTE5: + return PipelineType::PIPE_MTE5; + case pto::PIPE::PIPE_V2: + return PipelineType::PIPE_V2; + case pto::PIPE::PIPE_FIX: + return PipelineType::PIPE_FIX; + case pto::PIPE::VIRTUAL_PIPE_MTE2_L1A: + return PipelineType::VIRTUAL_PIPE_MTE2_L1A; + case pto::PIPE::VIRTUAL_PIPE_MTE2_L1B: + return PipelineType::VIRTUAL_PIPE_MTE2_L1B; + case pto::PIPE::PIPE_NUM: + case pto::PIPE::PIPE_UNASSIGNED: + return PipelineType::PIPE_UNASSIGNED; + } + return PipelineType::PIPE_UNASSIGNED; +} + +/// Mirrors `verifyBufSyncOp` (PTO.cpp): the attribute may be a plain PipeAttr, or a +/// pipe_event_type/sync_op_type that must be mapped. Declared in the header because +/// G1's coverage walk decodes the same attribute and must agree with this. +pto::PIPE mlir::pto::oracle::bufSyncPipe(mlir::Attribute opTypeAttr) { + if (!opTypeAttr) + return pto::PIPE::PIPE_UNASSIGNED; + if (auto pipeAttr = dyn_cast(opTypeAttr)) + return pipeAttr.getPipe(); + auto opTypeOr = parseSyncOpTypeLikeAttr(opTypeAttr); + if (succeeded(opTypeOr)) + return mapSyncOpTypeToPipe(*opTypeOr); + return pto::PIPE::PIPE_UNASSIGNED; +} + +llvm::SmallVector +mlir::pto::oracle::extractFromIR(func::FuncOp func) { + llvm::SmallVector records; + + func.walk([&](Operation *op) { + IRSyncRecord rec; + + auto setPipes = [&](pto::PIPE src, pto::PIPE dst) { + rec.srcPipe = stringifyPIPE(src).str(); + rec.dstPipe = stringifyPIPE(dst).str(); + rec.srcPipeType = toPipelineType(src); + rec.dstPipeType = toPipelineType(dst); + }; + + if (auto setFlag = dyn_cast(op)) { + rec.opName = "pto.set_flag"; + setPipes(setFlag.getSrcPipeAttr().getPipe(), + setFlag.getDstPipeAttr().getPipe()); + rec.eventId = static_cast(setFlag.getEventIdAttr().getEvent()); + } else if (auto waitFlag = dyn_cast(op)) { + rec.opName = "pto.wait_flag"; + setPipes(waitFlag.getSrcPipeAttr().getPipe(), + waitFlag.getDstPipeAttr().getPipe()); + rec.eventId = static_cast(waitFlag.getEventIdAttr().getEvent()); + } else if (auto getBuf = dyn_cast(op)) { + rec.opName = "pto.get_buf"; + rec.bufId = static_cast(getBuf.getBufId()); + rec.mode = static_cast(getBuf.getMode()); + rec.block = op->getBlock(); + pto::PIPE pipe = bufSyncPipe(getBuf.getOpTypeAttr()); + setPipes(pipe, pipe); + } else if (auto rlsBuf = dyn_cast(op)) { + rec.opName = "pto.rls_buf"; + rec.bufId = static_cast(rlsBuf.getBufId()); + rec.mode = static_cast(rlsBuf.getMode()); + rec.block = op->getBlock(); + pto::PIPE pipe = bufSyncPipe(rlsBuf.getOpTypeAttr()); + setPipes(pipe, pipe); + } else if (auto setDyn = dyn_cast(op)) { + // Rotating hazards emit these. They must be recorded, or a gate reading the + // emitted ops sees fewer sync ops than the kernel has. + // + // `eventId` stays -1 because the id is a RUNTIME value chosen by the selector + // chain rather than an attribute. THE ID ITSELF IS THEREFORE UNCHECKED: no gate + // here validates a rotating id, so a collision between two rotating hazards + // would not be reported. Recovering the ids statically is possible -- the + // selector is a chain of `arith.select` over constants -- and is not attempted. + rec.opName = "pto.set_flag_dyn"; + setPipes(setDyn.getSrcPipeAttr().getPipe(), + setDyn.getDstPipeAttr().getPipe()); + } else if (auto waitDyn = dyn_cast(op)) { + rec.opName = "pto.wait_flag_dyn"; + setPipes(waitDyn.getSrcPipeAttr().getPipe(), + waitDyn.getDstPipeAttr().getPipe()); + } else if (auto barrier = dyn_cast(op)) { + rec.opName = "pto.barrier"; + pto::PIPE pipe = barrier.getPipeAttr().getPipe(); + setPipes(pipe, pipe); + } else { + return WalkResult::advance(); + } + + rec.op = op; + rec.order = records.size(); + records.push_back(std::move(rec)); + return WalkResult::advance(); + }); + + return records; +} + +llvm::SmallVector +mlir::pto::oracle::extractFromSyncOps(const SyncOperations &ops) { + llvm::SmallVector records; + + for (const auto &group : ops) { + for (const auto &owned : group) { + const SyncOperation *sync = owned.get(); + if (!sync) + continue; + SyncOpRecord rec; + rec.type = SyncOperation::TypeName(sync->GetType()); + rec.typeCode = static_cast(sync->GetType()); + rec.mechanism = SyncOperation::MechanismName(sync->GetMechanism()); + rec.srcPipe = static_cast(sync->GetActualSrcPipe()); + rec.dstPipe = static_cast(sync->GetActualDstPipe()); + for (int id : sync->eventIds) + rec.eventIds.push_back(id); + rec.syncIndex = sync->GetSyncIndex(); + rec.irIndex = sync->GetSyncIRIndex(); + rec.compensationOf = sync->compensationOf; + // DERIVED, not copied from `SyncOperation::isCompensation`: that flag also + // steers codegen placement and is deliberately left unset (see + // synthesizeLoopCompensation). `compensationOf >= 0` is the oracle-only marker. + rec.isCompensation = sync->compensationOf >= 0; + records.push_back(std::move(rec)); + } + } + + return records; +} + + +void mlir::pto::oracle::printIRRecords(llvm::raw_ostream &os, + llvm::StringRef funcName, + llvm::ArrayRef records) { + os << "[sync-extract:ir] func=" << funcName << " records=" << records.size() + << "\n"; + for (const IRSyncRecord &rec : records) { + os << " #" << rec.order << " " << rec.opName + << " src=" << (rec.srcPipe.empty() ? "-" : rec.srcPipe) + << " dst=" << (rec.dstPipe.empty() ? "-" : rec.dstPipe); + if (rec.eventId >= 0) + os << " event_id=" << rec.eventId; + if (rec.bufId >= 0) + os << " buf_id=" << rec.bufId; + os << "\n"; + } +} + +void mlir::pto::oracle::printSyncOpRecords( + llvm::raw_ostream &os, llvm::StringRef funcName, + llvm::ArrayRef records) { + os << "[sync-extract:syncops] func=" << funcName + << " records=" << records.size() << "\n"; + for (const SyncOpRecord &rec : records) { + os << " #" << rec.syncIndex << " " << rec.type + << " mech=" << rec.mechanism + << " src=" + << pipelineTypeName(static_cast(rec.srcPipe)) + << " dst=" << pipelineTypeName(static_cast(rec.dstPipe)) + << " irIdx=" << rec.irIndex << " event_ids=["; + for (size_t i = 0; i < rec.eventIds.size(); ++i) { + if (i) + os << ","; + os << rec.eventIds[i]; + } + os << "]\n"; + } +} + +namespace { + +/// Renders the IR view of a function's emitted sync ops. Read-only. +struct PTODumpSyncExtractPass + : public mlir::pto::impl::PTODumpSyncExtractBase { + void runOnOperation() override { + func::FuncOp func = getOperation(); + if (func.isDeclaration()) + return; + auto records = oracle::extractFromIR(func); + oracle::emitReport([&](llvm::raw_ostream &os) { + oracle::printIRRecords(os, func.getSymName(), records); + }); + } +}; + +} // namespace + +std::unique_ptr mlir::pto::createPTODumpSyncExtractPass() { + return std::make_unique(); +} diff --git a/lib/PTO/Transforms/InsertSync/SyncOracleGates.cpp b/lib/PTO/Transforms/InsertSync/SyncOracleGates.cpp new file mode 100644 index 0000000000..916630332b --- /dev/null +++ b/lib/PTO/Transforms/InsertSync/SyncOracleGates.cpp @@ -0,0 +1,2622 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===- SyncOracleGates.cpp - Oracle correctness gates ---------------------===// +// +// Implements G3 (device-id legality), G2 (non-interference), the happens-before +// edge model, and G1-self (dependency coverage), plus the three read-only passes +// that run them on a compiled function. +//===----------------------------------------------------------------------===// + +#include "PTO/Transforms/InsertSync/SyncOracleGates.h" +#include "llvm/Support/xxhash.h" +#include "PTO/Transforms/InsertSync/SyncMacroModel.h" +#include "PTO/Transforms/Passes.h" +#include "PTO/IR/PTO.h" +#include "PTO/IR/PTOSyncUtils.h" +#include "PTO/Transforms/InsertSync/MemoryDependentAnalyzer.h" +#include "PTO/Transforms/InsertSync/PTOIRTranslator.h" +#include "llvm/ADT/DenseMap.h" +#include +#include "llvm/Support/MemoryBuffer.h" +#include "llvm/ADT/DenseSet.h" +#include "llvm/ADT/MapVector.h" +#include "llvm/Support/Format.h" +#include "mlir/Dialect/Arith/IR/Arith.h" +#include "mlir/Dialect/SCF/IR/SCF.h" +#include "mlir/IR/Matchers.h" +#include "mlir/Interfaces/LoopLikeInterface.h" +#include "mlir/Interfaces/SideEffectInterfaces.h" +#include +#include + +namespace mlir { +namespace pto { +namespace func = ::mlir::func; + +#define GEN_PASS_DEF_PTOCHECKSYNCIDS +#define GEN_PASS_DEF_PTOCHECKSYNCSELFCOVERAGE +#define GEN_PASS_DEF_PTOCHECKSYNCINTERFERENCE +#define GEN_PASS_DEF_PTODUMPSYNCCOVERAGE +#define GEN_PASS_DEF_PTOCHECKSYNCCOVERAGE +#define GEN_PASS_DEF_PTODUMPSYNCCOUNTS +#define GEN_PASS_DEF_PTOCHECKSYNCCOUNTFLOOR +#include "PTO/Transforms/Passes.h.inc" + +} // namespace pto +} // namespace mlir + +using namespace mlir; +using namespace mlir::pto; +using namespace mlir::pto::oracle; + +llvm::StringRef mlir::pto::oracle::idViolationKindName(IdViolationKind kind) { + switch (kind) { + case IdViolationKind::EventIdOutOfRange: + return "event-id-out-of-range"; + case IdViolationKind::EventIdReservedForDirection: + return "event-id-reserved-for-direction"; + case IdViolationKind::EventIdReservedForBlockAll: + return "event-id-reserved-for-block-all"; + case IdViolationKind::BlockAllEventIdWrong: + return "block-all-event-id-wrong"; + case IdViolationKind::BarrierCarriesEventId: + return "barrier-carries-event-id"; + case IdViolationKind::BufIdOutOfRange: + return "buf-id-out-of-range"; + case IdViolationKind::BufIdExceedsArchCapacity: + return "buf-id-exceeds-arch-capacity"; + case IdViolationKind::EventIdCollidesWithMacroHidden: + return "event-id-collides-with-macro-hidden"; + case IdViolationKind::EventSetWaitUnbalanced: + return "event-set-wait-unbalanced"; + } + return "?"; +} + +namespace { + +std::string pipeDir(PipelineType src, PipelineType dst) { + return (oracle::pipelineTypeName(src) + "->" + oracle::pipelineTypeName(dst)) + .str(); +} + +/// R1 + R2: an intra-core set/wait event id. +void checkIntraCoreEventId(int64_t id, PipelineType src, PipelineType dst, + unsigned poolSize, unsigned order, + llvm::StringRef opName, + llvm::SmallVectorImpl &out) { + if (id < 0 || id >= static_cast(poolSize)) { + out.push_back({IdViolationKind::EventIdOutOfRange, order, opName.str(), id, + ("intra-core event pool for " + pipeDir(src, dst) + " is [0, " + + std::to_string(poolSize) + ")")}); + return; + } + uint64_t reserved = SyncEventIdAllocation::GetReservedEventIdNum(src, dst); + if (reserved > 0 && id >= static_cast(poolSize - reserved)) { + out.push_back( + {IdViolationKind::EventIdReservedForDirection, order, opName.str(), id, + (pipeDir(src, dst) + " reserves its top " + std::to_string(reserved) + + " id(s); available is [0, " + std::to_string(poolSize - reserved) + + ")")}); + } +} + +/// R4 + R5: a buffer id. +void checkBufId(int64_t id, unsigned order, llvm::StringRef opName, + const DeviceIdLimits &limits, + llvm::SmallVectorImpl &out) { + if (id < 0 || id >= static_cast(limits.bufIdHwMax)) { + out.push_back({IdViolationKind::BufIdOutOfRange, order, opName.str(), id, + ("hardware buf_id field is [0, " + + std::to_string(limits.bufIdHwMax) + ")")}); + return; + } + if (id >= static_cast(limits.bufIdCapacity)) { + out.push_back({IdViolationKind::BufIdExceedsArchCapacity, order, + opName.str(), id, + ("this compilation exposes K=" + + std::to_string(limits.bufIdCapacity) + + " buffer id(s); buffer-id sync needs K > 0 (A5)")}); + } +} + +} // namespace + +llvm::SmallVector +mlir::pto::oracle::checkDeviceIdLegality(llvm::ArrayRef records, + const DeviceIdLimits &limits) { + llvm::SmallVector violations; + + for (const IRSyncRecord &rec : records) { + if (rec.opName == "pto.set_flag" || rec.opName == "pto.wait_flag") { + checkIntraCoreEventId(rec.eventId, rec.srcPipeType, rec.dstPipeType, + limits.intraCoreEventIdNum, rec.order, rec.opName, + violations); + } else if (rec.opName == "pto.get_buf" || rec.opName == "pto.rls_buf") { + checkBufId(rec.bufId, rec.order, rec.opName, limits, violations); + } else if (rec.opName == "pto.barrier" && rec.eventId >= 0) { + violations.push_back({IdViolationKind::BarrierCarriesEventId, rec.order, + rec.opName, rec.eventId, + "a pipe barrier takes no event id"}); + } + } + + // Every (direction, id) must carry as many sets as waits. A shortfall either way + // is a hang: a wait with no set never retires, and a set with no wait leaves the + // flag raised for whoever takes the id next. + // + // Rotating pairs are excluded, not overlooked: `set_flag_dyn`/`wait_flag_dyn` + // choose their id at runtime and it is recorded as -1, so a rotating set cannot be + // paired with its wait. Nothing here checks a rotating id. + llvm::DenseMap, std::pair> tally; + llvm::DenseMap, unsigned> firstOrder; + for (const IRSyncRecord &rec : records) { + bool isSet = rec.opName == "pto.set_flag"; + bool isWait = rec.opName == "pto.wait_flag"; + if ((!isSet && !isWait) || rec.eventId < 0) + continue; + auto key = std::make_tuple(static_cast(rec.srcPipeType), + static_cast(rec.dstPipeType), rec.eventId); + auto &slot = tally[key]; + (isSet ? slot.first : slot.second)++; + if (!firstOrder.count(key)) + firstOrder[key] = rec.order; + } + llvm::SmallVector> keys; + for (const auto &kv : tally) + keys.push_back(kv.first); + llvm::sort(keys); + for (const auto &key : keys) { + auto [sets, waits] = tally[key]; + if (sets == waits) + continue; + violations.push_back( + {IdViolationKind::EventSetWaitUnbalanced, firstOrder[key], "pto.set_flag", + static_cast(std::get<2>(key)), + ("direction " + std::to_string(std::get<0>(key)) + "->" + + std::to_string(std::get<1>(key)) + " carries " + std::to_string(sets) + + " set_flag and " + std::to_string(waits) + " wait_flag")}); + } + + return violations; +} + +llvm::SmallVector +mlir::pto::oracle::checkDeviceIdLegality(llvm::ArrayRef records, + const DeviceIdLimits &limits) { + using TYPE = SyncOperation::TYPE; + llvm::SmallVector violations; + + // The block-sync pool only shrinks when a SYNC_BLOCK_ALL is actually present; + // mirrors SyncEventIdAllocation::reserveBlockAllEventIds(). + bool blockAllPresent = llvm::any_of(records, [](const SyncOpRecord &rec) { + return rec.typeCode == static_cast(TYPE::SYNC_BLOCK_ALL); + }); + const unsigned blockPool = + limits.blockSyncEventIdNum - + (blockAllPresent ? kReservedBlockSyncEventIdNum : 0u); + + for (const SyncOpRecord &rec : records) { + auto src = static_cast(rec.srcPipe); + auto dst = static_cast(rec.dstPipe); + + switch (static_cast(rec.typeCode)) { + case TYPE::SET_EVENT: + case TYPE::WAIT_EVENT: + // EVERY id a hazard holds is checked, not just the first. A rotating hazard + // carries several, and its emitted form (`set_flag_dyn`/`wait_flag_dyn`) + // resolves the id at runtime, so once emitted the id cannot be checked at + // all -- these pre-codegen records are the only place the full set is visible. + // Weakening or narrowing this loop -- + // e.g. back to `eventIds[0]` -- would not fail any test: it would make that + // blind spot LIVE SILENTLY, because no other view can catch an illegal id + // on a rotating pair. + // + // Iterating the FULL set is the point: nothing else validates these ids, so + // narrowing it to `eventIds[0]` would silently stop checking the rest. + for (int id : rec.eventIds) + checkIntraCoreEventId(id, src, dst, limits.intraCoreEventIdNum, + rec.syncIndex, rec.type, violations); + break; + + case TYPE::SYNC_BLOCK_SET: + case TYPE::SYNC_BLOCK_WAIT: + for (int id : rec.eventIds) { + if (blockAllPresent && (id == static_cast(kBlockSyncAllCubeEventId) || + id == static_cast(kBlockSyncAllVectorEventId))) { + violations.push_back( + {IdViolationKind::EventIdReservedForBlockAll, rec.syncIndex, + rec.type, id, + ("ids " + std::to_string(kBlockSyncAllCubeEventId) + "/" + + std::to_string(kBlockSyncAllVectorEventId) + + " belong to sync_block_all while one is live")}); + continue; + } + if (id < 0 || id >= static_cast(blockPool)) + violations.push_back({IdViolationKind::EventIdOutOfRange, + rec.syncIndex, rec.type, id, + ("block-sync pool is [0, " + + std::to_string(blockPool) + ")")}); + } + break; + + case TYPE::SYNC_BLOCK_ALL: + for (int id : rec.eventIds) + if (id != static_cast(kBlockSyncAllCubeEventId) && + id != static_cast(kBlockSyncAllVectorEventId)) + violations.push_back( + {IdViolationKind::BlockAllEventIdWrong, rec.syncIndex, rec.type, + id, + ("sync_block_all must use " + + std::to_string(kBlockSyncAllCubeEventId) + " (cube) or " + + std::to_string(kBlockSyncAllVectorEventId) + " (vector)")}); + break; + + case TYPE::PIPE_BARRIER: + case TYPE::PIPE_BARRIER_CUBE: + case TYPE::PIPE_BARRIER_VECTOR: + for (int id : rec.eventIds) + violations.push_back({IdViolationKind::BarrierCarriesEventId, + rec.syncIndex, rec.type, id, + "a pipe barrier takes no event id"}); + break; + } + } + + return violations; +} + +llvm::SmallVector +mlir::pto::oracle::checkMacroHiddenEventCollisions( + const SyncIRs &syncIR, llvm::ArrayRef records) { + llvm::SmallVector violations; + + // One reservation per (direction, id) a macro's library implementation uses, + // over the macro's own span. Built to the same rule the incumbent reserves by, + // so that a green result here means what a green result there means. + struct Reservation { + int srcPipe, dstPipe, id; + unsigned begin, end; + std::string macroName; + }; + llvm::SmallVector reserved; + + for (size_t i = 0; i < syncIR.size(); ++i) { + auto *first = dyn_cast(syncIR[i].get()); + if (!first || first->macroOpInstanceId != 0 || !first->elementOp) + continue; + auto model = pto::getSyncMacroModel(first->elementOp); + if (!model || model->hiddenEvents.empty()) + continue; + + // The span is first phase -> last phase of the SAME op, then padded one index + // each side. Without the pad, a hazard ending exactly where the call begins + // reads as disjoint from it and the collision is missed. + unsigned end = first->GetIndex() + 1; + for (size_t j = i + 1; j < syncIR.size(); ++j) { + auto *other = dyn_cast(syncIR[j].get()); + if (!other || other->elementOp != first->elementOp) + continue; + end = other->GetIndex(); + } + unsigned begin = first->GetIndex(); + if (begin > 0) + --begin; + if (end + 1 < syncIR.size()) + ++end; + if (begin >= end) + continue; + + for (const auto &hidden : model->hiddenEvents) + for (unsigned id : hidden.eventIds) + reserved.push_back({int(hidden.srcPipe), int(hidden.dstPipe), int(id), + begin, end, + first->elementOp->getName().getStringRef().str()}); + } + if (reserved.empty()) + return violations; + + // An allocated hazard occupies the interval spanned by its own records: the set + // and the wait share a `syncIndex`, so the group's min/max irIndex is its live + // range. Barriers carry no id and cannot collide. + struct Live { + int srcPipe = -1, dstPipe = -1; + unsigned lo = ~0u, hi = 0; + llvm::SmallVector ids; + }; + llvm::MapVector byHazard; + for (const SyncOpRecord &r : records) { + if (r.eventIds.empty()) + continue; + Live &L = byHazard[r.syncIndex]; + L.srcPipe = r.srcPipe; + L.dstPipe = r.dstPipe; + L.lo = std::min(L.lo, r.irIndex); + L.hi = std::max(L.hi, r.irIndex); + for (int id : r.eventIds) + if (!llvm::is_contained(L.ids, id)) + L.ids.push_back(id); + } + + for (const auto &entry : byHazard) { + const Live &L = entry.second; + for (const Reservation &R : reserved) { + if (L.srcPipe != R.srcPipe || L.dstPipe != R.dstPipe) + continue; + if (L.hi < R.begin || R.end < L.lo) + continue; // disjoint spans + for (int id : L.ids) { + if (id != R.id) + continue; + violations.push_back( + {IdViolationKind::EventIdCollidesWithMacroHidden, entry.first, + "set/wait", id, + ("event id " + std::to_string(id) + " on direction " + + std::to_string(R.srcPipe) + "->" + std::to_string(R.dstPipe) + + " is reserved by `" + R.macroName + "` over SyncIR [" + + std::to_string(R.begin) + ", " + std::to_string(R.end) + + "]; this hazard holds it over [" + std::to_string(L.lo) + ", " + + std::to_string(L.hi) + + "], so the library's own wait can be satisfied by this producer")}); + } + } + } + return violations; +} + + +void mlir::pto::oracle::printIdViolations(llvm::raw_ostream &os, + llvm::StringRef funcName, + llvm::StringRef view, + size_t recordCount, + llvm::ArrayRef violations) { + os << "[sync-gate:g3] func=" << funcName << " view=" << view + << " records=" << recordCount << " violations=" << violations.size() + << (violations.empty() ? " OK" : "") << "\n"; + for (const IdViolation &v : violations) { + os << " !! #" << v.order << " " << v.opName + << " kind=" << idViolationKindName(v.kind) << " id=" << v.id << " : " + << v.detail << "\n"; + } +} + +//===----------------------------------------------------------------------===// +// G4: sync-count floor +//===----------------------------------------------------------------------===// + +llvm::StringRef mlir::pto::oracle::countViolationKindName(CountViolationKind k) { + switch (k) { + case CountViolationKind::TotalIncreased: + return "total-sync-count-increased"; + case CountViolationKind::BarrierIncreased: + return "barrier-count-increased"; + case CountViolationKind::BarrierAllIncreased: + return "barrier-all-count-increased"; + case CountViolationKind::MissingReference: + return "missing-reference-profile"; + } + return "?"; +} + +SyncCountProfile +mlir::pto::oracle::computeSyncCounts(llvm::ArrayRef records) { + SyncCountProfile p; + for (const IRSyncRecord &rec : records) { + // Rotation's dyn ops count toward the SAME classes as their static + // counterparts -- they are set/wait pairs whose id is chosen at runtime. They + // were previously counted as NOTHING, so G4 undercounted every rotating + // kernel and an allocator that replaced static pairs with dyn pairs would + // have looked like it REDUCED the op count. + if (rec.opName == "pto.set_flag" || rec.opName == "pto.set_flag_dyn") + ++p.setFlag; + else if (rec.opName == "pto.wait_flag" || rec.opName == "pto.wait_flag_dyn") + ++p.waitFlag; + else if (rec.opName == "pto.get_buf") + ++p.getBuf; + else if (rec.opName == "pto.rls_buf") + ++p.rlsBuf; + else if (rec.opName == "pto.barrier") { + ++p.barrier; + if (rec.srcPipeType == PipelineType::PIPE_ALL) + ++p.barrierAll; + } + } + return p; +} + +llvm::SmallVector +mlir::pto::oracle::checkSyncCountFloor(const SyncCountProfile &cand, + const SyncCountProfile &ref) { + llvm::SmallVector violations; + + if (cand.total() > ref.total()) + violations.push_back({CountViolationKind::TotalIncreased, cand.total(), + ref.total(), + "the new mode emits more sync ops than InsertSync"}); + + // Load-bearing: one barrier can replace a set/wait pair, lowering the total + // while strictly worsening the schedule. On vadd the degenerate ties on total. + if (cand.barrier > ref.barrier) + violations.push_back( + {CountViolationKind::BarrierIncreased, cand.barrier, ref.barrier, + "a barrier orders every pipe; a set/wait pair orders one direction"}); + + if (cand.barrierAll > ref.barrierAll) + violations.push_back({CountViolationKind::BarrierAllIncreased, + cand.barrierAll, ref.barrierAll, + "PIPE_ALL barriers fully serialize the core"}); + + return violations; +} + +void mlir::pto::oracle::printSyncCounts(llvm::raw_ostream &os, + llvm::StringRef funcName, + const SyncCountProfile &p) { + os << "[sync-count] func=" << funcName << " set=" << p.setFlag + << " wait=" << p.waitFlag << " get=" << p.getBuf << " rls=" << p.rlsBuf + << " barrier=" << p.barrier << " barrier_all=" << p.barrierAll + << " total=" << p.total() << "\n"; +} + +llvm::StringMap +mlir::pto::oracle::parseSyncCounts(llvm::StringRef text) { + llvm::StringMap profiles; + + llvm::SmallVector lines; + text.split(lines, '\n'); + for (llvm::StringRef line : lines) { + line = line.trim(); + if (!line.consume_front("[sync-count]")) + continue; + + llvm::StringRef name; + SyncCountProfile p; + llvm::SmallVector fields; + line.split(fields, ' ', /*MaxSplit=*/-1, /*KeepEmpty=*/false); + for (llvm::StringRef field : fields) { + auto [key, value] = field.split('='); + if (key == "func") { + name = value; + continue; + } + unsigned parsed = 0; + if (value.getAsInteger(10, parsed)) + continue; + if (key == "set") + p.setFlag = parsed; + else if (key == "wait") + p.waitFlag = parsed; + else if (key == "get") + p.getBuf = parsed; + else if (key == "rls") + p.rlsBuf = parsed; + else if (key == "barrier") + p.barrier = parsed; + else if (key == "barrier_all") + p.barrierAll = parsed; + // `total` is derived; ignore it on read so a hand-edited file cannot lie. + } + if (!name.empty()) + profiles[name] = p; + } + + return profiles; +} + +void mlir::pto::oracle::printCountViolations( + llvm::raw_ostream &os, llvm::StringRef funcName, + const SyncCountProfile &cand, const SyncCountProfile &ref, + llvm::ArrayRef violations) { + os << "[sync-gate:g4] func=" << funcName << " candidate_total=" << cand.total() + << " reference_total=" << ref.total() + << " violations=" << violations.size() << (violations.empty() ? " OK" : "") + << "\n"; + os << " candidate: "; + printSyncCounts(os, funcName, cand); + os << " reference: "; + printSyncCounts(os, funcName, ref); + for (const CountViolation &v : violations) { + os << " !! kind=" << countViolationKindName(v.kind) + << " candidate=" << v.candidate << " reference=" << v.reference << " : " + << v.detail << "\n"; + } +} + +//===----------------------------------------------------------------------===// +// G2: non-interference +//===----------------------------------------------------------------------===// + +llvm::StringRef mlir::pto::oracle::interferenceKindName(InterferenceKind k) { + switch (k) { + case InterferenceKind::EventIdNested: + return "event-id-nested"; + case InterferenceKind::EventIdWaitWithoutSet: + return "event-id-wait-without-set"; + case InterferenceKind::EventIdUnclosed: + return "event-id-unclosed"; + case InterferenceKind::EventIdPairGuardMismatch: + return "event-id-pair-guard-mismatch"; + case InterferenceKind::BufIdNested: + return "buf-id-nested"; + case InterferenceKind::BufIdReleaseWithoutAcquire: + return "buf-id-release-without-acquire"; + case InterferenceKind::BufIdUnclosed: + return "buf-id-unclosed"; + case InterferenceKind::BufIdPipeMismatch: + return "buf-id-pipe-mismatch"; + case InterferenceKind::BufIdSinglePipe: + return "buf-id-single-pipe"; + case InterferenceKind::BufIdTooManyPipes: + return "buf-id-spans-more-than-2-pipes"; + case InterferenceKind::BufIdPairSplitAcrossBlocks: + return "buf-id-pair-split-across-blocks"; + case InterferenceKind::BufIdDirectionalModeUnsupported: + return "buf-id-directional-mode-unsupported"; + } + return "?"; +} + +llvm::StringRef mlir::pto::oracle::severityName(Severity s) { + return s == Severity::Warning ? "warning" : "error"; +} + +namespace { + +/// One live get_buf, remembered so its release can be checked against it: the +/// release must be on the same pipe (B4) and in the same block (B7). +struct BufOpen { + unsigned order; + PipelineType pipe; + mlir::Block *block; +}; + +/// One live interval stack per id key. `depth > 1` means two hazards hold the +/// same id at once -- the bug G2 exists to find. +/// An open interval owned by nothing in particular. The IR view has no notion of an +/// owning hazard (a rotating pair's ops carry eventId = -1 there), so it passes this +/// and every overlap conflicts, exactly as before. +constexpr int kNoOwner = -1; + +/// The `scf.if`/`scf.index_switch` REGIONS enclosing `op`, outermost first. +/// +/// Loops are skipped on purpose. Two ops in different blocks of the same loop nest +/// are still reached together whenever the loop body runs at all, and requiring the +/// two halves of an event pair to share a block would reject the set-outside / +/// wait-inside priming idiom, which is what the corpus overwhelmingly emits. +llvm::SmallVector conditionalGuards(Operation *op) { + llvm::SmallVector guards; + for (Operation *p = op; p; p = p->getParentOp()) { + Operation *parent = p->getParentOp(); + if (parent && isa(parent)) + guards.push_back(p->getParentRegion()); + } + std::reverse(guards.begin(), guards.end()); + return guards; +} + +/// One open interval. `op` is null in the pre-codegen record view, which carries no +/// ops; a null disables only the checks that need a position in the IR. +struct Open { + unsigned order; + int owner; + Operation *op; +}; + +struct LiveIntervals { + llvm::SmallVector opens; + + /// Returns the order of an already-live interval that CONFLICTS with this open, or + /// -1 if this open is legal. + /// + /// Two opens with the same non-sentinel owner do NOT conflict: they are two halves + /// of ONE hazard -- the loop-carried compensation pair and the in-body pair it + /// primes, which share the rotating ids by construction (one stamp covers set, + /// wait, head and tail alike). Anything else still conflicts, INCLUDING a + /// compensation op against a different hazard, which is the whole point of keying on + /// the owner rather than just skipping compensation records. + int64_t open(unsigned order, int owner = kNoOwner, Operation *op = nullptr) { + int64_t live = -1; + for (const auto &o : opens) { + if (owner != kNoOwner && o.owner == owner) + continue; // same hazard: its own prime/drain, not a competing interval + live = int64_t(o.order); + break; + } + opens.push_back({order, owner, op}); + return live; + } + /// The interval this close pops, or nullopt when there is none to pop. + std::optional close() { + if (opens.empty()) + return std::nullopt; + Open top = opens.back(); + opens.pop_back(); + return top; + } +}; + +std::string eventKeyName(PipelineType src, PipelineType dst, int64_t id) { + return (oracle::pipelineTypeName(src) + "->" + oracle::pipelineTypeName(dst) + + " id=" + llvm::Twine(id)) + .str(); +} + +/// The shared scan. `Key` is whatever makes two intervals conflict. +template +void scanOpen(llvm::DenseMap &live, const KeyT &key, + unsigned order, llvm::StringRef opName, llvm::StringRef keyName, + InterferenceKind nestedKind, + llvm::SmallVectorImpl &out, + int owner = kNoOwner, Operation *op = nullptr) { + int64_t alreadyLive = live[key].open(order, owner, op); + if (alreadyLive >= 0) + out.push_back({nestedKind, Severity::Error, order, unsigned(alreadyLive), + opName.str(), keyName.str(), + "this id is still live from the interval opened earlier; two " + "overlapping hazards share it"}); +} + +template +void scanClose(llvm::DenseMap &live, const KeyT &key, + unsigned order, llvm::StringRef opName, llvm::StringRef keyName, + InterferenceKind orphanKind, + llvm::SmallVectorImpl &out, + Operation *op = nullptr) { + std::optional opened = live[key].close(); + if (!opened) { + out.push_back({orphanKind, Severity::Error, order, 0, opName.str(), + keyName.str(), "closes an id that no earlier op opened"}); + return; + } + // The two halves must be equally guarded, or a path runs one without the other. + // A flattened scan cannot see this: the counts balance and the stack pairs them. + if (op && opened->op && + conditionalGuards(opened->op) != conditionalGuards(op)) + out.push_back({InterferenceKind::EventIdPairGuardMismatch, Severity::Error, + order, opened->order, opName.str(), keyName.str(), + "this op and the one that opened the id sit under different " + "conditionals, so some path executes one without the other: a " + "wait with no set hangs, a set with no wait leaks the id"}); +} + +} // namespace + +llvm::SmallVector +mlir::pto::oracle::checkNonInterference(llvm::ArrayRef records) { + llvm::SmallVector violations; + + // Event ids are per-(src,dst) disjoint hardware; buffer ids are a global pool. + llvm::DenseMap, LiveIntervals> eventLive; + // Buffer ids need more than a depth counter: the release must match the + // acquire's pipe (B4) and block (B7), so remember both per open interval. + llvm::DenseMap> bufLive; + std::map> bufPipes; // ordered: deterministic reporting + + for (const IRSyncRecord &rec : records) { + // set_flag_dyn / wait_flag_dyn carry a runtime id, recorded as -1, so they + // never reach this check and a rotating id is not validated here. + + if (rec.opName == "pto.set_flag" || rec.opName == "pto.wait_flag") { + auto key = std::make_tuple(int(rec.srcPipeType), int(rec.dstPipeType), + rec.eventId); + std::string name = eventKeyName(rec.srcPipeType, rec.dstPipeType, rec.eventId); + if (rec.opName == "pto.set_flag") + scanOpen(eventLive, key, rec.order, rec.opName, name, + InterferenceKind::EventIdNested, violations, kNoOwner, rec.op); + else + scanClose(eventLive, key, rec.order, rec.opName, name, + InterferenceKind::EventIdWaitWithoutSet, violations, rec.op); + } else if (rec.opName == "pto.get_buf" || rec.opName == "pto.rls_buf") { + // Buffer-ID legality. See SyncOracleGates.h for the rules, and + // docs/bufid_sync_a5_design.md "Correctness Invariants" for the protocol. + std::string key = ("buf_id=" + llvm::Twine(rec.bufId)).str(); + + // Track pipes-per-id (B5/B6) for the bidirectional token only: the + // directional mode has a different pipe discipline and is deferred. + if (rec.mode == 0) + bufPipes[rec.bufId].insert(int(rec.srcPipeType)); + + // The strict pairing discipline is defined for the bidirectional token + // (mode 0). The directional mode is explicitly freed from strict pairing + // by the spec and is a deferred mechanism -- report it, do not pair it. + if (rec.mode != 0) { + violations.push_back( + {InterferenceKind::BufIdDirectionalModeUnsupported, Severity::Error, + rec.order, 0, rec.opName, key, + "mode " + std::to_string(rec.mode) + + " is the directional token (set_flagV2), a deferred mechanism " + "this allocator does not emit"}); + continue; + } + + if (rec.opName == "pto.get_buf") { + // B1: re-acquiring a live id. Covers same-pipe nesting and + // cross-pipe interleave alike -- both are "opened twice". + auto &live = bufLive[rec.bufId]; + if (!live.empty()) { + violations.push_back( + {InterferenceKind::BufIdNested, Severity::Error, rec.order, + live.front().order, rec.opName, key, + "this id is still live from the interval opened earlier; a " + "second acquire double-increments the counter and the get may " + "never pop"}); + } + live.push_back({rec.order, rec.srcPipeType, rec.block}); + } else { + auto &live = bufLive[rec.bufId]; + if (live.empty()) { + // B2 + violations.push_back( + {InterferenceKind::BufIdReleaseWithoutAcquire, Severity::Error, + rec.order, 0, rec.opName, key, + "closes an id that no earlier get_buf opened"}); + } else { + BufOpen open = live.back(); + live.pop_back(); + // B4: the release must be on the same pipe as its acquire. + if (open.pipe != rec.srcPipeType) { + violations.push_back( + {InterferenceKind::BufIdPipeMismatch, Severity::Error, + rec.order, open.order, rec.opName, key, + "released on " + + oracle::pipelineTypeName(rec.srcPipeType).str() + + " but acquired on " + + oracle::pipelineTypeName(open.pipe).str() + + "; the token brackets one op on one pipe"}); + } + // B7: the pair must sit in one block, or only one arm may execute. + if (open.block && rec.block && open.block != rec.block) { + violations.push_back( + {InterferenceKind::BufIdPairSplitAcrossBlocks, Severity::Error, + rec.order, open.order, rec.opName, key, + "get_buf and rls_buf are in different blocks; if they are in " + "different control-flow arms only one executes and the id's " + "counters desync"}); + } + } + } + } + } + + // B5 / B6: how many distinct pipes does each id span? + llvm::SmallVector ids; + for (const auto &entry : bufPipes) + ids.push_back(entry.first); + llvm::sort(ids); + for (int64_t id : ids) { + const auto &pipes = bufPipes[id]; + std::string key = ("buf_id=" + llvm::Twine(id)).str(); + if (pipes.size() == 1) { + violations.push_back( + {InterferenceKind::BufIdSinglePipe, Severity::Error, 0, 0, + "pto.get_buf", key, + "spans only 1 pipe (" + + oracle::pipelineTypeName(PipelineType(*pipes.begin())).str() + + "); buffer-ID is only for sync between TWO different pipes -- " + "same-pipe ordering needs bar.pipe"}); + } else if (pipes.size() > 2) { + // Soft rule, not an illegality: this gate must accept an id that spans + // more than two pipes, so the case is reported rather than failed. + violations.push_back( + {InterferenceKind::BufIdTooManyPipes, Severity::Warning, 0, 0, + "pto.get_buf", key, + "spans " + std::to_string(pipes.size()) + + " pipes (>2 pipe-pairs on one id); the spec advises against " + "this -- it is the coalescing bound the allocator must decide"}); + } + } + + // A leaked id: opened, never closed. Deterministic order for diffability. + llvm::SmallVector leaks; + for (const auto &entry : eventLive) + for (const auto &open : entry.second.opens) { + unsigned order = open.order; + leaks.push_back({InterferenceKind::EventIdUnclosed, Severity::Error, + order, order, "pto.set_flag", + eventKeyName(PipelineType(std::get<0>(entry.first)), + PipelineType(std::get<1>(entry.first)), + std::get<2>(entry.first)), + "opened but never waited: the id is leaked"}); + } + for (const auto &entry : bufLive) + for (const BufOpen &open : entry.second) + leaks.push_back({InterferenceKind::BufIdUnclosed, Severity::Error, + open.order, open.order, "pto.get_buf", + ("buf_id=" + llvm::Twine(entry.first)).str(), + "acquired but never released: the id's counters desync"}); + llvm::sort(leaks, [](const InterferenceViolation &a, + const InterferenceViolation &b) { + return a.order < b.order; + }); + violations.append(leaks.begin(), leaks.end()); + + return violations; +} +llvm::SmallVector +mlir::pto::oracle::checkNonInterference(llvm::ArrayRef records) { + using TYPE = SyncOperation::TYPE; + llvm::SmallVector violations; + llvm::DenseMap, LiveIntervals> eventLive; + + // THE SCAN REQUIRES PROGRAM ORDER, AND THE CALLER'S ORDER IS NOT IT. + // + // `extractFromSyncOps` emits records in STORAGE order: group by group, and + // within a group the set before the wait -- regardless of where they actually + // sit in the program. For a BACKWARD-matched pair (a loop-carried WAR, whose + // wait is satisfied by the previous iteration's set and therefore appears + // EARLIER in the program than its set) that hands the scan [set, wait], which + // reads as perfectly well-formed. The gate then reports OK on a kernel whose + // event realization deadlocks on iteration 1 -- 188 such pairs exist across 49 + // of 98 corpus kernels. + // + // Sorting by `irIndex` restores true program order for the set/wait relation. + // Safe to do here, and verified across the corpus: NO pair has + // set.irIndex == wait.irIndex (644 forward, 188 backward, 0 ties), so the + // relation is never ambiguous. + // + // KNOWN LIMIT, deliberately not papered over: `irIndex` is SyncIR-ELEMENT + // granular, not op granular -- two syncs on the same element (one in + // `pipeBefore`, one in `pipeAfter`) share an index, and `SyncOpRecord` does not + // record which side. A stable sort leaves those in storage order. That cannot + // affect the set/wait relation (0 ties, above), but it means the relative order + // of two DIFFERENT hazards colliding on one element is not resolved here. It + // only matters once ids are reused across hazards, and G2 on the IR view + // -- true program order, post-codegen -- remains the authority. + llvm::SmallVector ordered(records.begin(), records.end()); + llvm::stable_sort(ordered, [](const SyncOpRecord &a, const SyncOpRecord &b) { + return a.irIndex < b.irIndex; + }); + + for (const SyncOpRecord &rec : ordered) { + TYPE type = static_cast(rec.typeCode); + if (type != TYPE::SET_EVENT && type != TYPE::WAIT_EVENT) + continue; + for (int id : rec.eventIds) { + auto key = std::make_tuple(rec.srcPipe, rec.dstPipe, int64_t(id)); + std::string name = eventKeyName(PipelineType(rec.srcPipe), + PipelineType(rec.dstPipe), id); + // The OWNING hazard: a compensation pair reports the in-body hazard it primes, + // everything else reports itself. So a compensation op and the pair it primes + // share an owner and do not nest, while a collision with any other hazard -- + // compensation or not -- is still reported. Scoping this to `isCompensation` + // alone would have excused the latter too, and syncops is the ONLY view that + // checks rotating ids at all (the IR view's dyn ops carry eventId = -1). + int owner = rec.isCompensation && rec.compensationOf >= 0 + ? rec.compensationOf + : int(rec.syncIndex); + if (type == TYPE::SET_EVENT) + scanOpen(eventLive, key, rec.syncIndex, rec.type, name, + InterferenceKind::EventIdNested, violations, owner); + else + scanClose(eventLive, key, rec.syncIndex, rec.type, name, + InterferenceKind::EventIdWaitWithoutSet, violations); + } + } + + llvm::SmallVector leaks; + for (const auto &entry : eventLive) + for (const auto &open : entry.second.opens) { + unsigned order = open.order; + leaks.push_back({InterferenceKind::EventIdUnclosed, Severity::Error, + order, order, "set_event", + eventKeyName(PipelineType(std::get<0>(entry.first)), + PipelineType(std::get<1>(entry.first)), + std::get<2>(entry.first)), + "opened but never waited: the id is leaked"}); + } + llvm::sort(leaks, [](const InterferenceViolation &a, + const InterferenceViolation &b) { + return a.order < b.order; + }); + violations.append(leaks.begin(), leaks.end()); + + return violations; +} + + +unsigned mlir::pto::oracle::countErrors( + llvm::ArrayRef violations) { + return llvm::count_if(violations, [](const InterferenceViolation &v) { + return v.severity == Severity::Error; + }); +} + +void mlir::pto::oracle::printInterferenceViolations( + llvm::raw_ostream &os, llvm::StringRef funcName, llvm::StringRef view, + size_t recordCount, llvm::ArrayRef violations) { + unsigned errors = countErrors(violations); + unsigned warnings = violations.size() - errors; + os << "[sync-gate:g2] func=" << funcName << " view=" << view + << " records=" << recordCount << " errors=" << errors + << " warnings=" << warnings << (errors == 0 ? " OK" : "") << "\n"; + for (const InterferenceViolation &v : violations) { + os << (v.severity == Severity::Warning ? " ~~ " : " !! "); + os << "#" << v.order << " " << v.opName + << " kind=" << interferenceKindName(v.kind) << " " << v.key; + if (v.kind == InterferenceKind::EventIdNested || + v.kind == InterferenceKind::BufIdNested || + v.kind == InterferenceKind::BufIdPipeMismatch || + v.kind == InterferenceKind::BufIdPairSplitAcrossBlocks) + os << " live_since=#" << v.openedAt; + os << " : " << v.detail << "\n"; + } +} + +//===----------------------------------------------------------------------===// +// The happens-before edge model, consumed by G1-self +//===----------------------------------------------------------------------===// + +namespace { + +/// An op that needs ordering: carries a pipe and touches memory. This is the +/// same test InjectBarrierAllSync uses to decide where a barrier is required. +bool isAnchor(Operation *op) { + if (isa(op)) + return false; + auto memEffect = dyn_cast(op); + if (!memEffect || !(memEffect.hasEffect() || + memEffect.hasEffect())) + return false; + return isa(op); +} + +/// A sync op, positioned by how many anchors precede it. +struct SyncEvent { + enum Kind { + BarrierAll, BarrierPipe, SetFlag, WaitFlag, GetBuf, RlsBuf, + SetFlagDyn, WaitFlagDyn + } kind; + unsigned slot; + pto::PIPE srcPipe; + pto::PIPE dstPipe; + int64_t id; // event id, or buf id + // The iteration dimension. `order` is the walk position, used only to say + // "before"/"after" in program order; `loop` is the NEAREST enclosing loop, or + // null at function level. + Operation *op = nullptr; + unsigned order = 0; + Operation *loop = nullptr; + /// Rotation depth N for a Dyn op: the ordering it establishes is at DISTANCE N, + /// not 1. 1 for every static op. + unsigned rotation = 1; +}; + +/// What one loop contributes to the iteration dimension: the anchor range of +/// its body (inclusive) and the walk position of the earliest op inside it. +/// +/// `firstOrder` is taken from the loop's CONTENTS, not from the loop op itself: +/// `walk` is post-order, so the loop op is visited *after* its body, and using +/// its own position would place the whole body "before" the loop. +struct LoopSpan { + unsigned firstAnchor = std::numeric_limits::max(); + unsigned lastAnchor = 0; + unsigned firstOrder = std::numeric_limits::max(); + bool hasAnchors() const { return firstAnchor <= lastAnchor; } +}; + +/// Recover the rotation depth N from a `set_flag_dyn` / `wait_flag_dyn` id operand. +/// +/// Codegen builds the id as `eventIds[slot % N]`: an `arith.remui , N` +/// feeding a chain of `arith.select`s over N constants +/// (`CreateSetWaitOpForMultiBuffer`). N is the modulus, so walking back from the +/// operand to the nearest `RemUIOp` with a constant RHS recovers it exactly. +/// +/// Returns 0 when N cannot be recovered. Callers treat that as "unknown depth" +/// and credit NOTHING -- crediting an ordering whose distance cannot be named +/// would be a false pass, and this model has no way to express "some distance". +unsigned rotationDepthOf(Value eventId) { + llvm::SmallVector work{eventId}; + llvm::SmallPtrSet seen; + while (!work.empty()) { + Value v = work.pop_back_val(); + Operation *def = v.getDefiningOp(); + if (!def || !seen.insert(def).second) + continue; + if (auto rem = dyn_cast(def)) { + llvm::APInt n; + if (matchPattern(rem.getRhs(), m_ConstantInt(&n)) && n.getZExtValue() >= 2) + return unsigned(n.getZExtValue()); + } + for (Value operand : def->getOperands()) + work.push_back(operand); + } + return 0; +} + +/// Is `op` nested anywhere inside `loop`? +bool insideLoop(Operation *op, Operation *loop) { + for (Operation *p = op->getParentOp(); p; p = p->getParentOp()) + if (p == loop) + return true; + return false; +} + +/// The blocks on `loop`'s ancestor spine: the block holding `loop` itself, plus +/// the block holding each of its ancestor ops, up to the function. +/// +/// An op in one of these blocks and earlier in program order executes on EVERY +/// path that reaches `loop` (structured control flow, single-block regions), so +/// the spine is what "unconditionally before the loop" means concretely. An op +/// in any other block sits inside a region the loop is not inside -- a +/// conditional arm, or a sibling loop that may run zero times -- and may not +/// execute at all. This is strictly stronger than "not inside the loop", and it +/// has to be: see the P1' note in `isPrimed`. +llvm::SmallPtrSet spineBlocks(Operation *loop) { + llvm::SmallPtrSet blocks; + for (Operation *p = loop; p; p = p->getParentOp()) + if (Block *block = p->getBlock()) + blocks.insert(block); + return blocks; +} + +/// Does `op` execute exactly once per iteration of `loop`? +/// +/// Only if it is a DIRECT child of the loop's body: an op one level deeper sits +/// inside some other region -- an `scf.if` arm, or a nested loop that may run +/// zero times -- and may be skipped on an iteration. A carried pair whose +/// producer can be skipped establishes nothing on the iteration that skips it. +bool executesEachIteration(Operation *op, Operation *loop) { + return op->getParentOp() == loop; +} + +} // namespace + +CoverageProfile mlir::pto::oracle::computeCoverage( + func::FuncOp func, llvm::DenseMap *anchorIndex) { + CoverageProfile profile; + + llvm::SmallVector anchorPipes; + /// Parallel to `anchorPipes`, same index space, filled by the same walk. The edge + /// filter needs the op itself to ask whether a sync reaches it; building this list + /// in a second walk is what would let the two drift out of step. + llvm::SmallVector anchorOps; + llvm::SmallVector events; + std::string signatureText; + // Deterministic iteration order: the carried-edge scan walks these. + llvm::MapVector loops; + unsigned walkOrder = 0; + + // Enclosing loops of `op`, innermost first, stopping at the function. + auto enclosingLoops = [&](Operation *op) { + llvm::SmallVector chain; + for (Operation *p = op->getParentOp(); p && p != func.getOperation(); + p = p->getParentOp()) + if (isa(p)) + chain.push_back(p); + return chain; + }; + + func.walk([&](Operation *op) { + unsigned order = walkOrder++; + llvm::SmallVector chain = enclosingLoops(op); + for (Operation *loop : chain) { + LoopSpan &span = loops[loop]; + span.firstOrder = std::min(span.firstOrder, order); + } + + if (isAnchor(op)) { + pto::PIPE pipe = cast(op).getPipe(); + unsigned index = anchorPipes.size(); + anchorPipes.push_back(pipe); + anchorOps.push_back(op); + if (anchorIndex) + (*anchorIndex)[op] = index; + for (Operation *loop : chain) { + LoopSpan &span = loops[loop]; + span.firstAnchor = std::min(span.firstAnchor, index); + span.lastAnchor = std::max(span.lastAnchor, index); + } + signatureText += op->getName().getStringRef().str(); + signatureText += ':'; + signatureText += stringifyPIPE(pipe).str(); + // The carried-edge comparison presumes both runs agree on loop + // structure. The signature is where presumptions get checked rather than + // assumed, so the nest depth goes in it. + signatureText += "#"; + signatureText += std::to_string(chain.size()); + signatureText += ';'; + return; + } + unsigned slot = anchorPipes.size(); + Operation *nearest = chain.empty() ? nullptr : chain.front(); + if (auto setFlag = dyn_cast(op)) { + events.push_back({SyncEvent::SetFlag, slot, + setFlag.getSrcPipeAttr().getPipe(), + setFlag.getDstPipeAttr().getPipe(), + int64_t(setFlag.getEventIdAttr().getEvent()), op, order, + nearest}); + } else if (auto waitFlag = dyn_cast(op)) { + events.push_back({SyncEvent::WaitFlag, slot, + waitFlag.getSrcPipeAttr().getPipe(), + waitFlag.getDstPipeAttr().getPipe(), + int64_t(waitFlag.getEventIdAttr().getEvent()), op, order, + nearest}); + } else if (auto getBuf = dyn_cast(op)) { + // A spelling this does not decode drops the event, and a dropped event is + // indistinguishable from an absent one: the ordering the token establishes + // then reads as uncovered. `isConcreteSyncPipe` mirrors `verifyBufSyncOp`, + // which rejects PIPE_ALL and PIPE_UNASSIGNED on these ops. + pto::PIPE pipe = oracle::bufSyncPipe(getBuf.getOpTypeAttr()); + if (pto::isConcreteSyncPipe(pipe)) + events.push_back({SyncEvent::GetBuf, slot, pipe, pipe, + int64_t(getBuf.getBufId()), op, order, nearest}); + } else if (auto rlsBuf = dyn_cast(op)) { + pto::PIPE pipe = oracle::bufSyncPipe(rlsBuf.getOpTypeAttr()); + if (pto::isConcreteSyncPipe(pipe)) + events.push_back({SyncEvent::RlsBuf, slot, pipe, pipe, + int64_t(rlsBuf.getBufId()), op, order, nearest}); + } else if (auto setDyn = dyn_cast(op)) { + // The rotating pair is what multi-buffering emits, and it was + // previously invisible: the type chain below had no Dyn case, so the op + // fell off the end and contributed NO edge. `id` is -1 because there is no + // static id -- the id is chosen at runtime -- so it cannot collide with a + // static key. + events.push_back({SyncEvent::SetFlagDyn, slot, + setDyn.getSrcPipeAttr().getPipe(), + setDyn.getDstPipeAttr().getPipe(), -1, op, order, + nearest, rotationDepthOf(setDyn.getEventId())}); + } else if (auto waitDyn = dyn_cast(op)) { + events.push_back({SyncEvent::WaitFlagDyn, slot, + waitDyn.getSrcPipeAttr().getPipe(), + waitDyn.getDstPipeAttr().getPipe(), -1, op, order, + nearest, rotationDepthOf(waitDyn.getEventId())}); + } else if (auto barrier = dyn_cast(op)) { + pto::PIPE pipe = barrier.getPipeAttr().getPipe(); + if (pipe == pto::PIPE::PIPE_ALL) { + ++profile.barrierAll; + } + events.push_back({pipe == pto::PIPE::PIPE_ALL ? SyncEvent::BarrierAll + : SyncEvent::BarrierPipe, + slot, pipe, pipe, -1, op, order, nearest}); + } + }); + + profile.anchorCount = anchorPipes.size(); + // xxh3, NOT llvm::hash_value: this value crosses a process boundary, and + // llvm::hash_value seeds itself from a function address when + // LLVM_ENABLE_ABI_BREAKING_CHECKS is on, so ASLR makes identical source hash + // differently in every run. + profile.anchorSignature = llvm::xxh3_64bits(llvm::StringRef(signatureText)); + profile.signatureVersion = kCoverageSignatureVersion; + + llvm::DenseSet> edges; + /// `reachedBy` is the op whose execution the edge depends on -- the CONSUMING end of + /// the mechanism: a barrier itself, the `wait_flag` of a flag pair, the `get_buf` of + /// a token pair. An edge is credited only when that op is reached on every path that + /// reaches the sink, because a sync the sink can skip establishes nothing. + /// + /// "Reached on every path" is tested as: the conditionals enclosing the sync must be + /// a PREFIX of those enclosing the sink. Enclosing LOOPS are not compared. + /// + /// NOT `DominanceInfo`, and the difference is not cosmetic. Structural dominance + /// says an op in a loop body never dominates an op after the loop, which withdraws + /// credit from a barrier inside a constant-trip-count loop -- it always executes, so + /// the ordering does hold. Measured: that rejection alone turns + /// `control_flow_nested_vec` from clean to three uncovered same-pipe dependencies. + /// A loop whose trip count can be zero is a real hole, and it is NOT covered here; + /// the carried rules test the loop spine for that case instead. + /// + /// Sink side only, deliberately. A sync inside an `scf.if` arm legitimately orders a + /// dependence whose sink is in that same arm -- every path reaching the sink passes + /// through it -- so also requiring the sync to be reached whenever the SOURCE is + /// would reject the shape the emitter produces most. + auto addEdges = [&](unsigned beforeSlot, pto::PIPE fromPipe, unsigned afterSlot, + pto::PIPE toPipe, bool anyPipe, Operation *reachedBy) { + llvm::SmallVector syncGuards; + if (reachedBy) + syncGuards = conditionalGuards(reachedBy); + for (unsigned i = 0; i < beforeSlot; ++i) { + if (!anyPipe && anchorPipes[i] != fromPipe) + continue; + for (unsigned j = afterSlot; j < anchorPipes.size(); ++j) { + if (!anyPipe && anchorPipes[j] != toPipe) + continue; + if (reachedBy) { + llvm::SmallVector sinkGuards = + conditionalGuards(anchorOps[j]); + if (syncGuards.size() > sinkGuards.size() || + !std::equal(syncGuards.begin(), syncGuards.end(), + sinkGuards.begin())) + continue; + } + if (i < j) + edges.insert({i, j}); + } + } + }; + + for (size_t e = 0; e < events.size(); ++e) { + const SyncEvent &ev = events[e]; + switch (ev.kind) { + case SyncEvent::BarrierAll: + addEdges(ev.slot, ev.srcPipe, ev.slot, ev.dstPipe, /*anyPipe=*/true, ev.op); + break; + case SyncEvent::BarrierPipe: + addEdges(ev.slot, ev.srcPipe, ev.slot, ev.dstPipe, /*anyPipe=*/false, ev.op); + break; + case SyncEvent::SetFlag: + // Pair with the next matching wait. G2 guarantees the interval discipline + // that makes "next" the right answer. + for (size_t w = e + 1; w < events.size(); ++w) { + const SyncEvent &wait = events[w]; + if (wait.kind == SyncEvent::WaitFlag && wait.srcPipe == ev.srcPipe && + wait.dstPipe == ev.dstPipe && wait.id == ev.id) { + // The WAIT is the consuming end: the sink is ordered only if the wait ran. + addEdges(ev.slot, ev.srcPipe, wait.slot, wait.dstPipe, false, wait.op); + break; + } + } + break; + case SyncEvent::RlsBuf: + // Buffer-token happens-before: get_buf acquires what the previous holder + // released, so rls precedes get for the same buffer id. + for (size_t g = e + 1; g < events.size(); ++g) { + const SyncEvent &get = events[g]; + if (get.kind == SyncEvent::GetBuf && get.id == ev.id) + // The acquire is the consuming end, as the wait is for a flag pair. + addEdges(ev.slot, ev.srcPipe, get.slot, get.dstPipe, false, get.op); + } + break; + case SyncEvent::WaitFlag: + case SyncEvent::GetBuf: + break; + case SyncEvent::SetFlagDyn: + case SyncEvent::WaitFlagDyn: + // Deliberately NO FORWARD EDGE. A rotating pair orders iteration j-N + // against iteration j; within ONE iteration it orders nothing, because the + // set and the wait in the same iteration touch DIFFERENT event ids + // (eventIds[j%N] is set at the bottom, and the wait at the top consumed + // whatever iteration j-N set). Crediting a same-iteration edge here would + // invent an ordering the hardware does not establish. Their contribution is + // the CARRIED edge at distance N, added by the carried scan below. + break; + } + } + + profile.edges.assign(edges.begin(), edges.end()); + llvm::sort(profile.edges); + + //===--------------------------------------------------------------------===// + // The iteration dimension. Rules 1-3 (see SyncOracleGates.h). + // + // Deliberately a SECOND, INDEPENDENT scan. The forward pass above already + // consumes the head/tail compensation into two forward pairs, so bending it to + // pair backwards would change nothing -- which is precisely the measurement + // that motivated this. Nothing above this line is touched, so no existing + // forward edge can move. + // + // BARRIER-REALISED CARRIED ORDERINGS are handled here: a barrier + // are credited by the covering rule below. It was going to be deferred + // as untestable, but adversarial review showed it FIRES TODAY -- see the + // derivation at the covering loop. + //===--------------------------------------------------------------------===// + + // (from, to, distance) -> the loop that carries it. Second insert from a + // DIFFERENT loop means one anchor pair is carried at two nest levels, which + // this flat key cannot tell apart. The counter is a tripwire: nonzero means the + // key has become ambiguous and the closure below cannot be trusted. + std::map, Operation *> carriedOwner; + unsigned loopKeyCollisions = 0; + + auto addCarried = [&](const SyncEvent &producer, const SyncEvent &consumer, + Operation *loop, unsigned distance) { + const LoopSpan &span = loops[loop]; + if (!span.hasAnchors()) + return; + // Rule 2. Both endpoints restricted to the carrying loop's body: an anchor + // outside it has no iteration index, so a d=1 edge touching one would be + // meaningless. + for (unsigned x = span.firstAnchor; + x <= span.lastAnchor && x < producer.slot; ++x) { + if (anchorPipes[x] != producer.srcPipe) + continue; + for (unsigned y = std::max(span.firstAnchor, consumer.slot); + y <= span.lastAnchor; ++y) { + if (anchorPipes[y] != consumer.dstPipe) + continue; + auto key = std::make_tuple(x, y, distance); + auto it = carriedOwner.find(key); + if (it == carriedOwner.end()) + carriedOwner[key] = loop; + else if (it->second != loop) + ++loopKeyCollisions; + } + } + }; + + // Rule 3: is this loop-carried EVENT pair primed by a set outside the loop? + // + // P1' IS A REACHABILITY TEST, NOT A NESTING TEST -- and that distinction was + // a REAL FALSE PASS, found by adversarial review and reproduced. The first + // version asked only `!insideLoop(head.op, loop)`, which credits a head that + // may never execute. Four deadlocking shapes were blessed `violations=0 OK`: + // (a) `scf.if %c { set_flag }` before the loop -- hangs when %c is false; + // (b) `scf.if %c { set_flag } else { }` -- prime and loop on + // MUTUALLY EXCLUSIVE arms, so the prime provably never ran when the loop + // runs: hangs for EVERY value of %c; + // (c) the same with a guarded drain too, which balances G2's flat scan -- + // every gate reported OK on a certain hang; + // (d) `scf.for %j = 0 to %n { set_flag }` before the loop -- a zero-trip + // prime, on the very kernel named `test_dynamic_valid_shape`. + // `scf.if` is not a LoopLikeOpInterface, so nesting-only reasoning cannot see + // any of them. Blessing a hang is worse than having no iteration dimension at + // all, so P1' requires the head to be ON THE LOOP'S SPINE. + auto isPrimed = [&](const SyncEvent &producer, Operation *loop) { + unsigned entry = loops[loop].firstOrder; + llvm::SmallPtrSet spine = spineBlocks(loop); + for (const SyncEvent &head : events) { + if (head.kind != SyncEvent::SetFlag || head.id != producer.id || + head.srcPipe != producer.srcPipe || head.dstPipe != producer.dstPipe) + continue; + // P1' -- unconditionally reached. Subsumes the old P1: a block inside the + // loop body is not on the spine either. + if (!spine.contains(head.op->getBlock())) + continue; + if (head.order >= entry) // P2 + continue; + bool consumed = false; // P3: still live at loop entry? + for (const SyncEvent &wait : events) { + if (wait.kind != SyncEvent::WaitFlag || wait.id != producer.id || + wait.srcPipe != producer.srcPipe || wait.dstPipe != producer.dstPipe) + continue; + if (insideLoop(wait.op, loop)) + continue; + if (wait.order > head.order && wait.order < entry) { + consumed = true; + break; + } + } + if (!consumed) + return true; + } + return false; + }; + + for (const auto &entry : loops) { + Operation *loop = entry.first; + + // Rule 1: match forward within THIS body only; the leftovers wrap. + // Key: (src, dst, id) for flags; the buffer rule keys on the buffer id + // alone, matching the forward rule above. + std::map, llvm::SmallVector> + openProducers, strandedConsumers; + auto flagKey = [](const SyncEvent &e) { + return std::make_tuple(int(e.srcPipe), int(e.dstPipe), e.id); + }; + auto bufKey = [](const SyncEvent &e) { + return std::make_tuple(-1, -1, e.id); + }; + + for (unsigned i = 0; i < events.size(); ++i) { + const SyncEvent &ev = events[i]; + if (ev.loop != loop) + continue; + // The mirror of P1' on the IN-BODY side, and a reproduced false pass in + // its own right: with an unconditional prime and drain but the in-body + // producer under `scf.if`, iterations 2..N have nothing to wait on and the + // kernel hangs -- yet G1 and G2 both reported OK. `ev.loop` is the NEAREST + // enclosing LOOP, so an op inside an `scf.if` inside the body still lands + // here; only a direct child of the body runs every iteration. + if (!executesEachIteration(ev.op, loop)) + continue; + switch (ev.kind) { + case SyncEvent::SetFlag: + case SyncEvent::SetFlagDyn: + openProducers[flagKey(ev)].push_back(i); + break; + case SyncEvent::RlsBuf: + openProducers[bufKey(ev)].push_back(i); + break; + case SyncEvent::WaitFlag: + case SyncEvent::WaitFlagDyn: + case SyncEvent::GetBuf: { + auto key = + ev.kind == SyncEvent::GetBuf ? bufKey(ev) : flagKey(ev); + auto &open = openProducers[key]; + if (!open.empty()) + open.erase(open.begin()); // forward pair, consumed in this iteration + else + strandedConsumers[key].push_back(i); + break; + } + case SyncEvent::BarrierAll: + case SyncEvent::BarrierPipe: + break; + } + } + + // Barrier-realised carried orderings fire on real kernels. A barrier in the + // body establishes carried orderings without any flag pair, so a candidate + // that serializes instead of pairing has nothing for Rule 1 to detect and + // was REJECTED for an ordering it does establish. That is a false failure, + // and it is reachable with an in-tree flag on an in-tree sample: + // --enable-inject-barrier-all-sync on samples/Sync/test_dynamic_valid_shape.pto + // reported `!! missing-backward-ordering anchor#1 -> anchor#0 (d=1)` against + // the InsertSync reference. (The earlier "no corpus kernel emits that shape" + // reading was scoped to PINNED tests, which is not the same as reachable.) + // + // THE COVERING RULE. Unroll two iterations around a body barrier at slot s: + // iter k: a_0 .. a_{s-1} BARRIER a_s .. a_{n-1} + // iter k+1: a_0 .. a_{s-1} BARRIER a_s .. a_{n-1} + // For the carried edge (from=x, to=y) -- "x in iter k before y in iter k+1": + // x < s => iter k's OWN barrier follows x and precedes all of iter k+1; + // y >= s => iter k+1's barrier precedes y and follows all of iter k. + // So it is covered iff (x < s || y >= s), i.e. UNCOVERED only in the wrap + // gap y < s <= x. A PIPE_ALL barrier covers any pipes; a per-pipe barrier + // covers only anchors on its own pipe, mirroring the forward rule. + // + // A barrier needs no priming -- it is unconditional hardware serialization, + // not a flag -- but it must still RUN every iteration, hence the + // `executesEachIteration` filter above applies to it as well. + for (const SyncEvent &ev : events) { + if (ev.loop != loop || !executesEachIteration(ev.op, loop)) + continue; + if (ev.kind != SyncEvent::BarrierAll && ev.kind != SyncEvent::BarrierPipe) + continue; + const LoopSpan &span = loops[loop]; + if (!span.hasAnchors()) + continue; + bool anyPipe = ev.kind == SyncEvent::BarrierAll; + for (unsigned x = span.firstAnchor; x <= span.lastAnchor; ++x) { + if (!anyPipe && anchorPipes[x] != ev.srcPipe) + continue; + for (unsigned y = span.firstAnchor; y <= span.lastAnchor; ++y) { + if (!anyPipe && anchorPipes[y] != ev.dstPipe) + continue; + if (!(x < ev.slot || y >= ev.slot)) + continue; + // A barrier drains the pipe, so it orders EVERY pair of adjacent + // iterations -- distance 1, and by transitive closure every larger + // distance too. It is the strongest carried ordering, so it is + // recorded at d=1 regardless of any rotation in the loop. + auto key = std::make_tuple(x, y, 1u); + if (!carriedOwner.count(key)) + carriedOwner[key] = loop; + } + } + } + + for (auto &open : openProducers) { + auto stranded = strandedConsumers.find(open.first); + if (stranded == strandedConsumers.end()) + continue; + unsigned pairs = std::min(open.second.size(), stranded->second.size()); + for (unsigned k = 0; k < pairs; ++k) { + const SyncEvent &producer = events[open.second[k]]; + const SyncEvent &consumer = events[stranded->second[k]]; + if (consumer.order >= producer.order) + continue; // not backward: nothing wraps + // Rule 3 applies to events only. A buffer token's ticket lock starts + // free, so iteration 1's get_buf needs no priming op -- there is no + // unprimed state to discriminate against. + // A DYN pair rotates: `wait_flag_dyn` in iteration j waits on + // eventIds[j % N] and `set_flag_dyn` in iteration k sets eventIds[k % N], + // so the wait at j is satisfied by the latest k < j with k%N == j%N -- + // i.e. k = j - N. The ordering it establishes is at DISTANCE N, not 1. + // Derived from the emitted selector's modulus, not assumed. + bool isDyn = producer.kind == SyncEvent::SetFlagDyn || + consumer.kind == SyncEvent::WaitFlagDyn; + unsigned distance = 1; + if (isDyn) { + unsigned n = std::max(producer.rotation, consumer.rotation); + if (n < 2) + continue; // depth unrecoverable: credit nothing, never guess + distance = n; + } + // Rule 3 (priming) applies to the static event pair only. A rotating pair + // is primed by N static set_flags before the loop -- emitted through the + // compensation fan-out -- which the static scan below already sees as + // ordinary primes on the same (src,dst). + if (producer.kind == SyncEvent::SetFlag && !isPrimed(producer, loop)) + continue; + addCarried(producer, consumer, loop, distance); + } + } + } + + for (const auto &owned : carriedOwner) + profile.carriedEdges.push_back({std::get<0>(owned.first), + std::get<1>(owned.first), + std::get<2>(owned.first)}); + llvm::sort(profile.carriedEdges); + + // Through emitReport like every other oracle print: nested func passes run in + // PARALLEL and ptoas does not disable threading, so an unsynchronised write here + // can interleave with another function's report. + if (loopKeyCollisions) + oracle::emitReport([&](llvm::raw_ostream &os) { + os << "[sync-cov] !! carried-edge-loop-collision func=" << func.getName() + << " count=" << loopKeyCollisions + << " : one anchor pair is carried at two nest levels; the " + "(from,to,distance) key cannot tell them apart\n"; + }); + + return profile; +} + +//===----------------------------------------------------------------------===// +// G1: coverage superset (differential) +//===----------------------------------------------------------------------===// + +llvm::StringRef +mlir::pto::oracle::coverageViolationKindName(CoverageViolationKind k) { + switch (k) { + case CoverageViolationKind::MissingOrdering: + return "missing-ordering"; + case CoverageViolationKind::MissingBackwardOrdering: + return "missing-backward-ordering"; + case CoverageViolationKind::AnchorMismatch: + return "anchor-signature-mismatch"; + case CoverageViolationKind::MissingReference: + return "missing-reference-profile"; + case CoverageViolationKind::ProfileVersionMismatch: + return "coverage-profile-version-mismatch"; + } + return "?"; +} + +llvm::SmallVector +mlir::pto::oracle::checkCoverageSuperset(const CoverageProfile &cand, + const CoverageProfile &ref) { + llvm::SmallVector violations; + + // Checked BEFORE the signature, because a stale profile's signature is guaranteed to + // differ and reporting that as an anchor mismatch would blame the kernel for a + // format change. + if (ref.signatureVersion != cand.signatureVersion) { + violations.push_back( + {CoverageViolationKind::ProfileVersionMismatch, 0, 0, + "the reference profile was written with signature version " + + std::to_string(ref.signatureVersion) + " but this build writes version " + + std::to_string(cand.signatureVersion) + + " -- regenerate it with --dump-sync-coverage; the stored digest is not " + "comparable with the one computed here"}); + return violations; + } + + if (cand.anchorCount != ref.anchorCount || + cand.anchorSignature != ref.anchorSignature) { + violations.push_back( + {CoverageViolationKind::AnchorMismatch, 0, 0, + "the two runs did not see the same ops: candidate has " + + std::to_string(cand.anchorCount) + " anchor(s), reference has " + + std::to_string(ref.anchorCount) + + " -- the coverage comparison would be meaningless"}); + return violations; + } + + llvm::DenseSet> have; + for (const auto &edge : cand.edges) + have.insert(edge); + for (const auto &edge : ref.edges) + if (!have.count(edge)) + violations.push_back( + {CoverageViolationKind::MissingOrdering, edge.first, edge.second, + "the reference orders this pair; the candidate does not"}); + + // Carried edges are a SEPARATE superset, not merged into the forward set. + // A carried edge (x, y, d=1) with x < y is possible and would key identically + // to the forward edge (x, y) -- yet it is a strictly WEAKER claim, so merging + // would let a candidate that established only the cross-iteration ordering + // satisfy a reference demanding the same-iteration one. + std::set> haveCarried; + for (const CarriedEdge &edge : cand.carriedEdges) + haveCarried.insert({edge.from, edge.to, edge.distance}); + for (const CarriedEdge &edge : ref.carriedEdges) + if (!haveCarried.count({edge.from, edge.to, edge.distance})) + violations.push_back( + {CoverageViolationKind::MissingBackwardOrdering, edge.from, edge.to, + "the reference orders this pair across iterations; the candidate " + "does not -- an unprimed loop-carried wait blocks forever", + edge.distance}); + + return violations; +} + +void mlir::pto::oracle::printCoverage(llvm::raw_ostream &os, + llvm::StringRef funcName, + const CoverageProfile &p) { + os << "[sync-cov] func=" << funcName << " anchors=" << p.anchorCount + << " sig=" << llvm::format_hex(p.anchorSignature, 18) + << " edge_count=" << p.edges.size() << " edges="; + for (size_t i = 0; i < p.edges.size(); ++i) { + if (i) + os << ","; + os << p.edges[i].first << "-" << p.edges[i].second; + } + // Carried-edge fields are APPENDED, never interleaved: every existing pin ends at the + // forward `edges=` list and FileCheck matches substrings, so a line that grows + // a suffix still matches. Reordering would break all of them for no reason. + os << " bedge_count=" << p.carriedEdges.size() << " bedges="; + for (size_t i = 0; i < p.carriedEdges.size(); ++i) { + if (i) + os << ","; + os << p.carriedEdges[i].from << "-" << p.carriedEdges[i].to << "@" + << p.carriedEdges[i].distance; + } + os << " barrier_all=" << p.barrierAll + // APPENDED, like the carried-edge fields above and for the same reason. + << " sigver=" << p.signatureVersion << "\n"; +} + +llvm::StringMap +mlir::pto::oracle::parseCoverage(llvm::StringRef text) { + llvm::StringMap profiles; + + llvm::SmallVector lines; + text.split(lines, '\n'); + for (llvm::StringRef line : lines) { + line = line.trim(); + if (!line.consume_front("[sync-cov]")) + continue; + + llvm::StringRef name; + CoverageProfile p; + llvm::SmallVector fields; + line.split(fields, ' ', -1, /*KeepEmpty=*/false); + for (llvm::StringRef field : fields) { + auto [key, value] = field.split('='); + if (key == "func") { + name = value; + } else if (key == "anchors") { + if (value.getAsInteger(10, p.anchorCount)) + p.anchorCount = 0; + } else if (key == "sig") { + if (value.getAsInteger(0, p.anchorSignature)) + p.anchorSignature = 0; + } else if (key == "sigver") { + // Absent in a profile from an older build, which leaves the member 0 and makes + // the version check below reject it. + if (value.getAsInteger(10, p.signatureVersion)) + p.signatureVersion = 0; + } else if (key == "edges" && !value.empty()) { + llvm::SmallVector pairs; + value.split(pairs, ',', -1, /*KeepEmpty=*/false); + for (llvm::StringRef pair : pairs) { + auto [a, b] = pair.split('-'); + unsigned from = 0, to = 0; + if (!a.getAsInteger(10, from) && !b.getAsInteger(10, to)) + p.edges.push_back({from, to}); + } + } else if (key == "barrier_all") { + // A profile written before this field existed parses to 0, which reads as + // "no barrier delta" and leaves the verdict untouched -- the field is + // reported, never differenced into a pass. + if (value.getAsInteger(10, p.barrierAll)) + p.barrierAll = 0; + } else if (key == "bedges" && !value.empty()) { + // `j-i@d`. A profile written before carried edges existed has no `bedges=` field and + // parses to an empty carried set -- which is the right reading for the + // hand-written deliberate-mismatch fixtures. No version tag: no .cov + // file is checked in, so producer and consumer always move together. + llvm::SmallVector triples; + value.split(triples, ',', -1, /*KeepEmpty=*/false); + for (llvm::StringRef triple : triples) { + auto [pair, distText] = triple.split('@'); + auto [a, b] = pair.split('-'); + unsigned from = 0, to = 0, distance = 0; + if (!a.getAsInteger(10, from) && !b.getAsInteger(10, to) && + !distText.getAsInteger(10, distance)) + p.carriedEdges.push_back({from, to, distance}); + } + } + } + if (!name.empty()) { + llvm::sort(p.edges); + llvm::sort(p.carriedEdges); + profiles[name] = p; + } + } + + return profiles; +} + +void mlir::pto::oracle::printCoverageViolations( + llvm::raw_ostream &os, llvm::StringRef funcName, + const CoverageProfile &cand, const CoverageProfile &ref, + llvm::ArrayRef violations) { + os << "[sync-gate:g1] func=" << funcName + << " candidate_edges=" << cand.edges.size() + << " reference_edges=" << ref.edges.size() + << " violations=" << violations.size() << (violations.empty() ? " OK" : "") + << " candidate_bedges=" << cand.carriedEdges.size() + << " reference_bedges=" << ref.carriedEdges.size() + << " candidate_barrier_all=" << cand.barrierAll + << " reference_barrier_all=" << ref.barrierAll << "\n"; + // The two barrier counts are reported and nothing is concluded from them. An + // earlier version added a line reading the difference as coarseness rather than + // lost coverage, and the deliberate-fault fixtures refused it: an empty allocation + // and a hand-written missing-carried-ordering both show the reference ahead on + // barriers, because the reference carries the auto-sync tail barrier and they do + // not. The same delta therefore appears on the exact fault class this gate exists + // to catch, and a line explaining it away would have excused them. The reading + // belongs where it can be qualified; see BARRIER COARSENESS in the header. + unsigned shown = 0; + for (const CoverageViolation &v : violations) { + if ((v.kind == CoverageViolationKind::MissingOrdering || + v.kind == CoverageViolationKind::MissingBackwardOrdering) && + shown >= 8) { + os << " ... and " << (violations.size() - shown) << " more\n"; + break; + } + ++shown; + os << " !! kind=" << coverageViolationKindName(v.kind); + if (v.kind == CoverageViolationKind::MissingOrdering) + os << " anchor#" << v.from << " -> anchor#" << v.to; + if (v.kind == CoverageViolationKind::MissingBackwardOrdering) + os << " anchor#" << v.from << " -> anchor#" << v.to << " (d=" << v.distance + << ")"; + os << " : " << v.detail << "\n"; + } +} + +//===----------------------------------------------------------------------===// +// G1-self: absolute coverage against the dependency analysis +//===----------------------------------------------------------------------===// + +namespace { + +/// The anchor pipe of a compound's op. Taken from `OpPipeInterface` -- the same +/// source `computeCoverage` keys its edge filtering on. +std::optional anchorPipe(Operation *op) { + if (auto pipeOp = dyn_cast(op)) + return pipeOp.getPipe(); + return std::nullopt; +} + +std::string anchorPipeName(Operation *op) { + if (auto pipe = anchorPipe(op)) + return stringifyPIPE(*pipe).str(); + return "?"; +} + +/// Can `a` and `b` both execute in one pass through the function? +/// +/// FALSE only when they sit in DIFFERENT REGIONS of a common `scf.if`: the arms +/// are mutually exclusive, so no ordering between them can ever be required and +/// no sync can ever establish one. This is the same reasoning the carried rules apply to +/// priming -- a set on the `then` arm cannot prime a loop on the `else` arm. +/// +/// DELIBERATELY AS NARROW AS THE CORPUS ALLOWS. A too-broad exclusivity +/// predicate silently EXCUSES real dependencies, and that failure is invisible in +/// any "the number went down" check -- so this recognises `scf.if` and nothing +/// else -- not `scf.index_switch`, `cf.cond_br` or `cf.switch`. If another +/// mutually-exclusive-region op appears in the post-sync IR, this must be revisited +/// explicitly rather than generalised on a hunch. +/// +/// A one-armed `scf.if` is handled correctly by construction: if `b` is outside +/// the `if` entirely, the `if` never appears as a common ancestor, so the pair is +/// NOT excluded -- `b` always runs while `a` runs conditionally, and the ordering +/// is still required. +bool canBothExecute(Operation *a, Operation *b) { + llvm::SmallDenseMap aRegionOf; + for (Operation *p = a; p && p->getParentOp(); p = p->getParentOp()) + aRegionOf[p->getParentOp()] = p->getParentRegion(); + for (Operation *p = b; p && p->getParentOp(); p = p->getParentOp()) { + auto it = aRegionOf.find(p->getParentOp()); + if (it == aRegionOf.end() || it->second == p->getParentRegion()) + continue; + if (isa(p->getParentOp())) + return false; + } + return true; +} + +/// The innermost loop enclosing `op`, or null. +Operation *innermostLoop(Operation *op) { + for (Operation *p = op->getParentOp(); p; p = p->getParentOp()) + if (isa(p)) + return p; + return nullptr; +} + +} // namespace + +mlir::pto::oracle::SelfCoverageReport +mlir::pto::oracle::computeSelfCoverage(func::FuncOp func) { + SelfCoverageReport report; + + // --- The CANDIDATE side: what the emitted sync actually orders ------------- + llvm::DenseMap anchorIndex; + CoverageProfile profile = computeCoverage(func, &anchorIndex); + unsigned n = profile.anchorCount; + + // Transitive closure with distances. reach[i][j] = 0 (same iteration), any + // distance up to the cap (loop-carried), or unreached. Happens-before is + // transitive and the raw edge set is not closed -- pipe filtering breaks chains + // across three pipes -- so closing it is required to avoid false "uncovered". + // Distances compose additively; only a sum above the cap is dropped. + // The closure is CAPPED, and the cap direction is load-bearing. + // + // `reach[i][j]` holds the SMALLEST distance at which emitted sync orders i + // before j. It used to be {0, 1, unreached}; rotation needs general N. The cap + // bounds the table, and ANYTHING ABOVE IT FALLS TO UNREACHED -- i.e. to + // UNCOVERED, never to covered. A cap that silently credited large N would be a + // false pass hiding in an implementation detail, which is the one direction + // that must not happen. + // + // 32 matches K, the buffer-id pool, and exceeds every event pool (8 per + // direction), so no representable rotation depth can reach it. + constexpr uint8_t kMaxDistance = 32; + constexpr uint8_t kUnreached = kMaxDistance + 1; + std::vector reach(size_t(n) * n, kUnreached); + auto at = [&](unsigned i, unsigned j) -> uint8_t & { + return reach[size_t(i) * n + j]; + }; + for (const auto &e : profile.edges) + if (e.first < n && e.second < n) + at(e.first, e.second) = 0; + for (const CarriedEdge &e : profile.carriedEdges) + if (e.from < n && e.to < n && e.distance >= 1 && + e.distance <= kMaxDistance) + at(e.from, e.to) = + std::min(at(e.from, e.to), uint8_t(e.distance)); + for (unsigned k = 0; k < n; ++k) + for (unsigned i = 0; i < n; ++i) { + uint8_t ik = at(i, k); + if (ik == kUnreached) + continue; + for (unsigned j = 0; j < n; ++j) { + uint8_t kj = at(k, j); + if (kj == kUnreached) + continue; + // Distances compose additively: i -before- k at d1 and k -before- j at + // d2 gives i -before- j at d1+d2. Above the cap the composition is + // DROPPED, which leaves the pair unreached -> uncovered. Conservative by + // construction. + unsigned sum = unsigned(ik) + unsigned(kj); + if (sum <= kMaxDistance) + at(i, j) = std::min(at(i, j), uint8_t(sum)); + } + } + + // --- The EXPECTED side: the front-end dependency analysis, used DIRECTLY --- + // Nothing below reads InsertSync's decisions. `IsMemInfoHasDependency` is + // deliberately NOT called: it layers InsertSync's policy (the tload->tload WAW + // exemption, the ACC read/read case) on top of DepBetween, and policy is the + // thing under audit. + // The A5 PIPE_V intra-pipe guarantee. Read through the SAME predicate the + // codegen uses (`isTargetArchA5`, PTO/IR/PTO.h) rather than re-deriving it from + // the module attribute, so the oracle and the emitter cannot drift apart. + const bool isA5 = pto::isTargetArchA5(func.getOperation()); + + MemoryDependentAnalyzer memAnalyzer; + SyncIRs syncIR; + Buffer2MemInfoMap buffer2MemInfoMap; + PTOIRTranslator translator(syncIR, memAnalyzer, buffer2MemInfoMap, func, + SyncAnalysisMode::NORMALSYNC); + translator.Build(); + + llvm::SmallVector compounds; + for (auto &element : syncIR) + if (auto *c = dyn_cast(element.get())) + if (c->elementOp) + compounds.push_back(c); + report.compounds = compounds.size(); + + auto anchorOf = [&](pto::CompoundInstanceElement *c) -> int { + auto it = anchorIndex.find(c->elementOp); + return it == anchorIndex.end() ? -1 : int(it->second); + }; + + // One requirement per (pair, hazard class, distance). + auto record = [&](pto::CompoundInstanceElement *src, + pto::CompoundInstanceElement *dst, llvm::StringRef hazard, + unsigned distance) { + int as = anchorOf(src), ad = anchorOf(dst); + if (as < 0 || ad < 0) { + // NOT dropped -- counted. An endpoint with no anchor cannot be expressed + // in the coverage coordinate system at all. + ++report.unmappable; + return; + } + ++report.dependencies; + // Any distance >= 1 is loop-carried; 0 is same-iteration. Counting only + // d == 1 would hide rotating deps from the report once they carry + // their true distance. + if (distance >= 1) + ++report.carriedDeps; + bool covered = at(unsigned(as), unsigned(ad)) != kUnreached && + at(unsigned(as), unsigned(ad)) <= distance; + if (covered) { + ++report.covered; + if (distance >= 1) + ++report.carriedCovered; + return; + } + // (a2) Mutually exclusive arms. Counted, NOT silently dropped: `armExcluded` + // exists so the exclusion is auditable, and so that + // `uncovered + armExcluded` reproduces the pre-predicate uncovered count + // exactly. A predicate that quietly swallowed real dependencies would look + // identical to a correct one in any total-only check. + if (!canBothExecute(src->elementOp, dst->elementOp)) { + ++report.armExcluded; + return; + } + // A5 PIPE_V intra-pipe ordering is treated as a TARGET GUARANTEE. No sync is + // emitted for such a pair on A5 (`SyncCodegen`, `LoweringSyncToPipe`), so + // demanding one would make this gate report every such pair on every A5 kernel. + // + // NARROW ON PURPOSE -- this is NOT "A5 same-pipe is fine". The uncovered A5 + // population is overwhelmingly V->V, but a residue is `MTE3->MTE3 waw/carried` + // and cross-pipe `MTE3->V war/carried`, which this test does not excuse. + // Widening it to "same pipe" would silence them -- re-blinding the gate in the + // same edit that fixes it. + // + // SCOPE. This test asks only whether both endpoints sit on PIPE_V; it does not + // distinguish a same-allocation pair from a cross-allocation one. Aliasing views + // are expressed as distinct allocations overlapping in memory once addresses are + // assigned, so the exemption is broader than a per-allocation test would be. + // + // It is deliberately not narrowed: the sync pass treats those cross-allocation + // pairs identically, excusing all of them and covering none, so narrowing here + // would reclassify them from excused to uncovered without changing what is + // emitted. + if (isA5) { + auto srcPipe = anchorPipe(src->elementOp); + auto dstPipe = anchorPipe(dst->elementOp); + if (srcPipe && dstPipe && *srcPipe == pto::PIPE::PIPE_V && + *dstPipe == pto::PIPE::PIPE_V) { + ++report.archGuaranteed; + return; + } + } + // (a3) PIPE_S->PIPE_S, on every arch. + // + // THIS IS INFERENCE FROM COMPILER BEHAVIOUR, NOT AN ARCHITECTURAL FACT, and it is + // counted separately from `archGuaranteed` for that reason. No ISA or design + // document in this tree states the ordering the scalar pipe gives its own ops, and + // `pipe_barrier(PIPE_S)` is expressible and does reach the emitted code, so the + // omission below cannot be argued from the barrier being unavailable. + // + // What it does rest on is the same assumption encoded independently TWICE: + // `IsNoNeedToInsertSync` returns early for a same-pipe PIPE_S pair before any + // dependence analysis runs, deliberately -- the comment beside it declines to + // short-circuit same-pipe pairs in general -- and GraphSyncSolver's same-pipe + // barrier path (`SyncSolver.cpp`, guarded by `corePipeSrc == corePipeDst`) returns + // for PIPE_S unconditionally while gating PIPE_V and PIPE_M on `isRegBasedArch`. + // Two separate allocators decline to order this pair on every arch, so demanding + // it here would make the gate unsatisfiable rather than strict. + // + // Arch-independent, unlike (a1), because that is how both allocators key it. + { + auto srcPipe = anchorPipe(src->elementOp); + auto dstPipe = anchorPipe(dst->elementOp); + if (srcPipe && dstPipe && *srcPipe == pto::PIPE::PIPE_S && + *dstPipe == pto::PIPE::PIPE_S) { + ++report.pipeSelfOrdered; + return; + } + } + if (anchorPipeName(src->elementOp) == anchorPipeName(dst->elementOp)) + ++report.uncoveredSamePipe; + else + ++report.uncoveredCrossPipe; + report.violations.push_back({unsigned(as), unsigned(ad), distance, + hazard.str(), + src->opName.getStringRef().str(), + dst->opName.getStringRef().str(), + anchorPipeName(src->elementOp), + anchorPipeName(dst->elementOp)}); + }; + + DepBaseMemInfoPairVec scratch; + // Accumulates the aliasing pairs across ALL hazard classes for one compound + // pair, because `scratch` is reset per call and the slot count below must see + // every pair, not just the last class tested. + DepBaseMemInfoPairVec allPairs; + auto dep = [&](pto::CompoundInstanceElement *a, + pto::CompoundInstanceElement *b, bool aDef, bool bDef) { + scratch.clear(); + bool found = memAnalyzer.DepBetween(aDef ? a->defVec : a->useVec, + bDef ? b->defVec : b->useVec, scratch); + if (found) + allPairs.append(scratch.begin(), scratch.end()); + return found; + }; + + for (size_t i = 0; i < compounds.size(); ++i) { + for (size_t j = i + 1; j < compounds.size(); ++j) { + pto::CompoundInstanceElement *front = compounds[i]; + pto::CompoundInstanceElement *now = compounds[j]; + + // Three hazard classes, front -> now, in the SAME iteration. + allPairs.clear(); + bool raw = dep(front, now, /*aDef=*/true, /*bDef=*/false); // W then R + bool war = dep(front, now, /*aDef=*/false, /*bDef=*/true); // R then W + bool waw = dep(front, now, /*aDef=*/true, /*bDef=*/true); // W then W + if (raw) + record(front, now, "raw", 0); + if (war) + record(front, now, "war", 0); + if (waw) + record(front, now, "waw", 0); + + // The SAME alias relation also creates a LOOP-CARRIED requirement when + // both sit in one loop body: `now` in iteration k conflicts with `front` in + // iteration k+1, across the back edge. A forward RAW (front writes, now + // reads) is a carried WAR (now reads, then front writes next iteration), + // and so on -- which is why the sync pass emits backward pairs at + // all. This is the requirement only carried edges can satisfy. + if (!(raw || war || waw)) + continue; + Operation *loop = innermostLoop(front->elementOp); + if (!loop || loop != innermostLoop(now->elementOp)) + continue; + + // THE CARRIED DISTANCE IS THE SLOT COUNT, NOT ALWAYS 1. + // + // With N multi-buffer slots, iteration k touches slot k%N, so the next + // access that can CONFLICT is the one that reuses that slot -- iteration + // k+N. At N=1 this is 1 and nothing changes, which is the whole corpus. + // + // WIDEN, NEVER DROP. The forward precedent + // (`isForwardDepDroppableBySlotAffine`) DROPS a slot-disjoint dep, because + // forward it needs no sync at all. Mirroring that here would EXCUSE the + // dependency, and a kernel with NO ROTATION would then pass -- the + // false-pass direction. InsertSync's own comment guards exactly this: + // "Back-edge deps stay untouched -- they still need per-slot syncing + // through the dyn-event-id pipeline." The dep is real; only its DISTANCE + // moves. There is deliberately no skip/continue on this path. + // + // INDEPENDENCE: `baseAddresses` is front-end data, populated by the + // translator from the program's own `pto.multi_tile_get` slot structure. It + // is a fact about the PROGRAM, not about anything InsertSync decided. + unsigned slots = 1; + for (const auto &pair : allPairs) { + if (pair.first) + slots = std::max(slots, unsigned(pair.first->baseAddresses.size())); + if (pair.second) + slots = std::max(slots, unsigned(pair.second->baseAddresses.size())); + } + if (raw) + record(now, front, "war/carried", slots); + if (war) + record(now, front, "raw/carried", slots); + if (waw) + record(now, front, "waw/carried", slots); + } + } + + // SELF-PAIRS: one compound against ITSELF across the back edge. + // + // The pair loop above starts at `j = i + 1`, so `x` in iteration k is never + // asked about against `x` in iteration k+1. That requirement is real -- an op + // that touches the same location on consecutive iterations must be ordered + // against its own previous execution -- and until it is enumerated here it sits + // in NO bucket: not `dependencies`, so not `covered`, `uncovered`, `armExcluded` + // or `archGuaranteed`. A gate that cannot see a requirement reports GREEN when + // the op realising it is deleted, which is indistinguishable from the + // requirement having never existed. + // + // The coverage side already answers these without change. A same-pipe body + // barrier records carried edges over every (x, y) anchor pair in its loop span + // subject to `x < slot || y >= slot`, which is trivially true when x == y, so + // the barrier contributes a carried SELF-edge at distance 1. Nothing seeds the + // closure diagonal otherwise -- the reach table starts every cell at + // `kUnreached` -- so a self-pair is covered exactly when some emitted op really + // does order the op against its own next execution. + // + // Only ops inside a loop are considered: without a back edge there is no second + // execution to conflict with. + for (size_t i = 0; i < compounds.size(); ++i) { + pto::CompoundInstanceElement *x = compounds[i]; + if (!innermostLoop(x->elementOp)) + continue; + + allPairs.clear(); + // Read against write in both orders: across the back edge "k writes then + // k+1 reads" and "k reads then k+1 writes" are two distinct orderings, and + // both have to hold. They are recorded separately for that reason, not by + // oversight. + bool waw = dep(x, x, /*aDef=*/true, /*bDef=*/true); + bool raw = dep(x, x, /*aDef=*/true, /*bDef=*/false); + bool war = dep(x, x, /*aDef=*/false, /*bDef=*/true); + if (!(waw || raw || war)) + continue; + + unsigned slots = 1; + for (const auto &pair : allPairs) { + if (pair.first) + slots = std::max(slots, unsigned(pair.first->baseAddresses.size())); + if (pair.second) + slots = std::max(slots, unsigned(pair.second->baseAddresses.size())); + } + + if (waw) + record(x, x, "waw/carried-self", slots); + if (raw) + record(x, x, "raw/carried-self", slots); + if (war) + record(x, x, "war/carried-self", slots); + } + + return report; +} + +void mlir::pto::oracle::printSelfCoverage(llvm::raw_ostream &os, + llvm::StringRef funcName, + const SelfCoverageReport &report, + unsigned cap) { + unsigned uncovered = report.violations.size(); + os << "[sync-gate:g1-self] func=" << funcName + << " compounds=" << report.compounds + << " deps=" << report.dependencies << " covered=" << report.covered + << " uncovered=" << uncovered << " (samepipe=" << report.uncoveredSamePipe + << " crosspipe=" << report.uncoveredCrossPipe << ")" + << " carried=" << report.carriedDeps << "/" + << report.carriedCovered << " arm_excluded=" << report.armExcluded + << " arch_guaranteed=" << report.archGuaranteed + << " pipe_self_ordered=" << report.pipeSelfOrdered + << " unmappable=" << report.unmappable + << (uncovered == 0 ? " OK" : "") << "\n"; + unsigned shown = 0; + for (const SelfCoverageViolation &v : report.violations) { + if (cap != 0 && shown >= cap) { + os << " ... and " << (uncovered - shown) << " more\n"; + break; + } + ++shown; + os << " !! kind=uncovered-dependency " << v.hazard << " anchor#" + << v.srcAnchor << " (" << v.srcName << ") -> anchor#" << v.dstAnchor + << " (" << v.dstName << ")" + << " " << v.srcPipe << "->" << v.dstPipe; + if (v.distance == 1) + os << " (d=1)"; + os << " : the dependency analysis reports this ordering; no emitted sync " + "establishes it\n"; + } +} + +namespace { + +/// Test-only synthetic faults for the classes the IR cannot spell. +/// +/// The injection happens *after* extraction, into the gate's own record buffer. +/// It never touches the IR and therefore cannot reach codegen. Its only purpose +/// is to prove the gate rejects what it claims to reject. +/// False when `kind` names no fault, which the caller must report: a silent no-op +/// would leave the gate finding nothing and passing, so a misspelled selector would +/// vacuously satisfy the very test that exists to prove the gate is not vacuous. +bool injectFaultRecords(llvm::StringRef kind, + llvm::SmallVectorImpl &irRecords, + llvm::SmallVectorImpl &syncOpRecords) { + using TYPE = SyncOperation::TYPE; + + if (kind == "event-oor") { + // An event id past the end of the intra-core pool. Unrepresentable in IR + // (`pto.set_flag`'s event_id enum stops at EVENT_ID7) but writable into + // SyncOperation::eventIds by any allocator. + SyncOpRecord rec; + rec.type = "set_event"; + rec.typeCode = static_cast(TYPE::SET_EVENT); + rec.srcPipe = static_cast(PipelineType::PIPE_V); + rec.dstPipe = static_cast(PipelineType::PIPE_MTE3); + rec.eventIds.push_back(9); + syncOpRecords.push_back(std::move(rec)); + return true; + } + + if (kind == "block-all-stomp") { + // A live sync_block_all owns id 14; a block set that also takes 14 stomps it. + SyncOpRecord all; + all.type = "sync_block_all"; + all.typeCode = static_cast(TYPE::SYNC_BLOCK_ALL); + all.srcPipe = static_cast(PipelineType::PIPE_ALL); + all.dstPipe = static_cast(PipelineType::PIPE_ALL); + all.eventIds.push_back(static_cast(kBlockSyncAllCubeEventId)); + syncOpRecords.push_back(std::move(all)); + + SyncOpRecord set; + set.type = "sync_block_set"; + set.typeCode = static_cast(TYPE::SYNC_BLOCK_SET); + set.srcPipe = static_cast(PipelineType::PIPE_MTE3); + set.dstPipe = static_cast(PipelineType::PIPE_V); + set.eventIds.push_back(static_cast(kBlockSyncAllCubeEventId)); + set.syncIndex = 1; + syncOpRecords.push_back(std::move(set)); + return true; + } + + if (kind == "bufid-oor") { + // Past the hardware buf_id field, which real IR cannot spell, so the fault is + // injected into the gate's record buffer instead. + IRSyncRecord rec; + rec.opName = "pto.get_buf"; + rec.srcPipe = "PIPE_MTE2"; + rec.dstPipe = "PIPE_MTE2"; + rec.srcPipeType = PipelineType::PIPE_MTE2; + rec.dstPipeType = PipelineType::PIPE_MTE2; + rec.bufId = 64; + rec.order = irRecords.size(); + irRecords.push_back(std::move(rec)); + return true; + } + + return false; +} + +/// Appends a synthetic G2 shape to a record list. Returns false when `kind` names +/// none, which the caller must report: injecting nothing would leave the gate with +/// nothing to find, so a misspelled selector would report clean and prove nothing. +/// +/// Both shapes concern the pre-codegen view, and neither can be written as input IR. +/// A compensation record's link to the hazard it primes exists only before codegen, +/// and a rotating hazard's ids are resolved at runtime once emitted. +bool injectInterferenceRecords(llvm::StringRef kind, + llvm::SmallVectorImpl &records) { + using TYPE = SyncOperation::TYPE; + + if (kind == "comp-conflict") { + // A compensation record shares its ids with the hazard it primes BY DESIGN, so + // that pairing must not read as a collision. A collision with any OTHER hazard + // still must: exempting on `isCompensation` alone would excuse both, which is why + // the scan keys on the owning hazard rather than on the flag. + SyncOpRecord comp; + comp.type = "set_event"; + comp.typeCode = static_cast(TYPE::SET_EVENT); + comp.srcPipe = static_cast(PipelineType::PIPE_MTE2); + comp.dstPipe = static_cast(PipelineType::PIPE_V); + comp.eventIds.push_back(3); + comp.syncIndex = 900; + comp.irIndex = 900; + comp.isCompensation = true; + comp.compensationOf = 901; + SyncOpRecord other = comp; + other.isCompensation = false; + other.compensationOf = -1; + other.syncIndex = 902; + other.irIndex = 901; + records.push_back(std::move(comp)); + records.push_back(std::move(other)); + return true; + } + + if (kind == "multi-id-conflict") { + // The scan must check EVERY id a hazard holds, not just the first. These two + // records differ on their first id and collide on their second, so a + // first-id-only scan finds nothing while the full scan reports the collision. + SyncOpRecord rot; + rot.type = "set_event"; + rot.typeCode = static_cast(TYPE::SET_EVENT); + rot.srcPipe = static_cast(PipelineType::PIPE_MTE2); + rot.dstPipe = static_cast(PipelineType::PIPE_V); + rot.eventIds.push_back(0); + rot.eventIds.push_back(1); + rot.syncIndex = 910; + rot.irIndex = 910; + rot.isCompensation = true; + rot.compensationOf = 911; + SyncOpRecord clash = rot; + clash.eventIds.clear(); + clash.eventIds.push_back(7); + clash.eventIds.push_back(1); + clash.isCompensation = false; + clash.compensationOf = -1; + clash.syncIndex = 912; + clash.irIndex = 911; + records.push_back(std::move(rot)); + records.push_back(std::move(clash)); + return true; + } + + return false; +} + +/// Runs G3 on a compiled function. Read-only apart from diagnostics. +struct PTOCheckSyncIdsPass + : public mlir::pto::impl::PTOCheckSyncIdsBase { + unsigned bufidCapacity = 0; + std::string injectFault; + + void runOnOperation() override { + func::FuncOp func = getOperation(); + if (func.isDeclaration()) + return; + + DeviceIdLimits limits; + limits.bufIdCapacity = bufidCapacity; + + llvm::SmallVector irRecords = oracle::extractFromIR(func); + llvm::SmallVector syncOpRecords; + if (!injectFault.empty() && + !injectFaultRecords(injectFault, irRecords, syncOpRecords)) { + func.emitError() << "unknown --check-sync-ids-inject-fault selector '" + << injectFault + << "'; expected event-oor, block-all-stomp or bufid-oor"; + signalPassFailure(); + return; + } + + auto irViolations = oracle::checkDeviceIdLegality(irRecords, limits); + llvm::SmallVector syncOpViolations; + if (!syncOpRecords.empty()) + syncOpViolations = oracle::checkDeviceIdLegality(syncOpRecords, limits); + + oracle::emitReport([&](llvm::raw_ostream &os) { + oracle::printIdViolations(os, func.getSymName(), "ir", irRecords.size(), + irViolations); + if (!syncOpRecords.empty()) + oracle::printIdViolations(os, func.getSymName(), "syncops", + syncOpRecords.size(), syncOpViolations); + }); + + size_t total = irViolations.size() + syncOpViolations.size(); + if (total > 0) { + func.emitError() << "G3 device-id legality: " << total + << " illegal sync id(s)"; + signalPassFailure(); + } + } +}; + +/// G2: runs the non-interference check on a compiled function. +struct PTOCheckSyncInterferencePass + : public mlir::pto::impl::PTOCheckSyncInterferenceBase< + PTOCheckSyncInterferencePass> { + std::string injectFault; + + void runOnOperation() override { + func::FuncOp func = getOperation(); + if (func.isDeclaration()) + return; + + auto records = oracle::extractFromIR(func); + auto violations = oracle::checkNonInterference(records); + oracle::emitReport([&](llvm::raw_ostream &os) { + oracle::printInterferenceViolations(os, func.getSymName(), "ir", + records.size(), violations); + }); + + // The two synthetic shapes concern the pre-codegen view, so they are judged on + // their own record list rather than mixed into the emitted-op scan above. + if (!injectFault.empty()) { + llvm::SmallVector synthetic; + if (!injectInterferenceRecords(injectFault, synthetic)) { + func.emitError() << "unknown --check-sync-interference-inject-fault " + "selector '" + << injectFault + << "'; expected comp-conflict or multi-id-conflict"; + signalPassFailure(); + return; + } + auto synthViolations = oracle::checkNonInterference(synthetic); + oracle::emitReport([&](llvm::raw_ostream &os) { + oracle::printInterferenceViolations(os, func.getSymName(), "syncops", + synthetic.size(), synthViolations); + }); + if (unsigned errors = oracle::countErrors(synthViolations)) { + func.emitError() << "G2 non-interference: " << errors + << " illegal sync id use(s)"; + signalPassFailure(); + return; + } + } + // Warnings (a soft rule, such as an id spanning more than two pipes) are + // reported but do not fail the gate; only errors do. + if (unsigned errors = oracle::countErrors(violations)) { + func.emitError() << "G2 non-interference: " << errors + << " illegal sync id use(s)"; + signalPassFailure(); + } + } +}; + +/// G1-self: ABSOLUTE, not differential. Asserts that every dependency the +/// front-end analysis reports is ordered by emitted sync. No reference file -- +/// the expected set comes from `DepBetween`, so there is nothing to compare +/// against and nothing inherited from InsertSync's choices. +struct PTOCheckSyncSelfCoveragePass + : public mlir::pto::impl::PTOCheckSyncSelfCoverageBase< + PTOCheckSyncSelfCoveragePass> { + /// Violations printed per function; 0 = all. Threaded in from the driver rather + /// than read from a global, so nothing that links this library inherits a flag. + unsigned maxViolations = 8; + + void runOnOperation() override { + func::FuncOp func = getOperation(); + if (func.isDeclaration()) + return; + auto report = oracle::computeSelfCoverage(func); + oracle::emitReport([&](llvm::raw_ostream &os) { + oracle::printSelfCoverage(os, func.getSymName(), report, maxViolations); + }); + if (!report.violations.empty()) { + func.emitError() << "G1-self coverage: " << report.violations.size() + << " dependency ordering(s) are not established by any " + "emitted sync"; + signalPassFailure(); + } + } +}; + + +/// G1 reference side: writes the happens-before edges this run's sync induces. +struct PTODumpSyncCoveragePass + : public mlir::pto::impl::PTODumpSyncCoverageBase { + void runOnOperation() override { + func::FuncOp func = getOperation(); + if (func.isDeclaration()) + return; + auto profile = oracle::computeCoverage(func); + oracle::emitReport([&](llvm::raw_ostream &os) { + oracle::printCoverage(os, func.getSymName(), profile); + }); + } +}; + +/// G1: every ordering the reference establishes must also be established here. +struct PTOCheckSyncCoveragePass + : public mlir::pto::impl::PTOCheckSyncCoverageBase { + std::string referenceFile; + + void runOnOperation() override { + func::FuncOp func = getOperation(); + if (func.isDeclaration()) + return; + + auto bufferOr = llvm::MemoryBuffer::getFile(referenceFile); + if (!bufferOr) { + func.emitError() << "G1 coverage: cannot read reference profile '" + << referenceFile << "': " << bufferOr.getError().message(); + return signalPassFailure(); + } + + llvm::StringMap reference = + oracle::parseCoverage((*bufferOr)->getBuffer()); + llvm::StringRef name = func.getSymName(); + CoverageProfile candidate = oracle::computeCoverage(func); + + auto it = reference.find(name); + if (it == reference.end()) { + oracle::emitReport([&](llvm::raw_ostream &os) { + os << "[sync-gate:g1] func=" << name << " violations=1\n !! kind=" + << oracle::coverageViolationKindName( + CoverageViolationKind::MissingReference) + << " : no reference profile for this function in '" << referenceFile + << "'\n"; + }); + func.emitError() << "G1 coverage: no reference profile for '" << name + << "'"; + return signalPassFailure(); + } + + const CoverageProfile &ref = it->second; + auto violations = oracle::checkCoverageSuperset(candidate, ref); + oracle::emitReport([&](llvm::raw_ostream &os) { + oracle::printCoverageViolations(os, name, candidate, ref, violations); + }); + + if (!violations.empty()) { + // An anchor mismatch is not "missing orderings" -- nothing is missing, the + // two runs simply did not compile the same program. Say so. + if (violations.front().kind == CoverageViolationKind::AnchorMismatch) + func.emitError() << "G1 coverage: candidate and reference did not see " + "the same program (anchor signature mismatch)"; + else + func.emitError() << "G1 coverage: " << violations.size() + << " ordering(s) the reference establishes are missing"; + signalPassFailure(); + } + } +}; + +/// Writes this run's kernel-body sync-op counts. Read-only. +struct PTODumpSyncCountsPass + : public mlir::pto::impl::PTODumpSyncCountsBase { + void runOnOperation() override { + func::FuncOp func = getOperation(); + if (func.isDeclaration()) + return; + auto profile = oracle::computeSyncCounts(oracle::extractFromIR(func)); + oracle::emitReport([&](llvm::raw_ostream &os) { + oracle::printSyncCounts(os, func.getSymName(), profile); + }); + } +}; + +/// G4: compares this run's counts against a reference run's profile. +struct PTOCheckSyncCountFloorPass + : public mlir::pto::impl::PTOCheckSyncCountFloorBase< + PTOCheckSyncCountFloorPass> { + std::string referenceFile; + + void runOnOperation() override { + func::FuncOp func = getOperation(); + if (func.isDeclaration()) + return; + + auto bufferOr = llvm::MemoryBuffer::getFile(referenceFile); + if (!bufferOr) { + func.emitError() << "G4 sync-count floor: cannot read reference profile '" + << referenceFile << "': " << bufferOr.getError().message(); + return signalPassFailure(); + } + + llvm::StringMap reference = + oracle::parseSyncCounts((*bufferOr)->getBuffer()); + + llvm::StringRef name = func.getSymName(); + SyncCountProfile candidate = + oracle::computeSyncCounts(oracle::extractFromIR(func)); + + auto it = reference.find(name); + if (it == reference.end()) { + // Not a pass: a gate that skips what it cannot find is not a gate. + oracle::emitReport([&](llvm::raw_ostream &os) { + os << "[sync-gate:g4] func=" << name << " violations=1\n !! kind=" + << oracle::countViolationKindName(CountViolationKind::MissingReference) + << " : no reference profile for this function in '" << referenceFile + << "'\n"; + }); + func.emitError() << "G4 sync-count floor: no reference profile for '" + << name << "'"; + return signalPassFailure(); + } + + const SyncCountProfile &ref = it->second; + auto violations = oracle::checkSyncCountFloor(candidate, ref); + oracle::emitReport([&](llvm::raw_ostream &os) { + oracle::printCountViolations(os, name, candidate, ref, violations); + }); + + if (!violations.empty()) { + func.emitError() << "G4 sync-count floor: " << violations.size() + << " regression(s) against the reference profile"; + signalPassFailure(); + } + } +}; +} // namespace + +std::unique_ptr +mlir::pto::createPTOCheckSyncIdsPass(unsigned bufidCapacity, + llvm::StringRef injectFault) { + auto pass = std::make_unique(); + pass->bufidCapacity = bufidCapacity; + pass->injectFault = injectFault.str(); + return pass; +} + +std::unique_ptr +mlir::pto::createPTOCheckSyncInterferencePass(llvm::StringRef injectFault) { + auto pass = std::make_unique(); + pass->injectFault = injectFault.str(); + return pass; +} + +std::unique_ptr +mlir::pto::createPTOCheckSyncSelfCoveragePass(unsigned maxViolations) { + auto pass = std::make_unique(); + pass->maxViolations = maxViolations; + return pass; +} + +std::unique_ptr mlir::pto::createPTODumpSyncCoveragePass() { + return std::make_unique(); +} + +std::unique_ptr +mlir::pto::createPTOCheckSyncCoveragePass(llvm::StringRef referenceFile) { + auto pass = std::make_unique(); + pass->referenceFile = referenceFile.str(); + return pass; +} + +std::unique_ptr mlir::pto::createPTODumpSyncCountsPass() { + return std::make_unique(); +} + +std::unique_ptr +mlir::pto::createPTOCheckSyncCountFloorPass(llvm::StringRef referenceFile) { + auto pass = std::make_unique(); + pass->referenceFile = referenceFile.str(); + return pass; +} diff --git a/lib/PTO/Transforms/InsertSync/UnifiedSyncModel.cpp b/lib/PTO/Transforms/InsertSync/UnifiedSyncModel.cpp new file mode 100644 index 0000000000..355a707a7f --- /dev/null +++ b/lib/PTO/Transforms/InsertSync/UnifiedSyncModel.cpp @@ -0,0 +1,972 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===- UnifiedSyncModel.cpp - Unified sync allocator data model -----------===// +// +// Builds the hazard set the allocator colors. Decides nothing. +// +//===----------------------------------------------------------------------===// + +#include "PTO/Transforms/InsertSync/UnifiedSyncModel.h" +#include "PTO/IR/PTO.h" +#include "mlir/IR/Matchers.h" +#include "PTO/Transforms/InsertSync/SyncMacroModel.h" +#include +#include +#include +#include "PTO/Transforms/InsertSync/SyncEventIdAllocation.h" +#include "PTO/Transforms/InsertSync/SyncOracleExtract.h" +#include "llvm/ADT/DenseSet.h" +#include "llvm/Support/Casting.h" + +using namespace mlir; +using namespace mlir::pto; +using namespace mlir::pto::unified; + +//===----------------------------------------------------------------------===// +// ResourceModel +//===----------------------------------------------------------------------===// + +unsigned ResourceModel::eventPoolSize(PipelineType src, PipelineType dst) const { + // Single source of truth, shared with the G3 gate: the allocator must not be + // able to believe in a bigger pool than the gate will accept. + uint64_t reserved = SyncEventIdAllocation::GetReservedEventIdNum(src, dst); + if (reserved >= kTotalEventIdNum) + return 0; + return static_cast(kTotalEventIdNum - reserved); +} + +//===----------------------------------------------------------------------===// +// Model construction +//===----------------------------------------------------------------------===// + +namespace { + +/// A direction key. Event flags are per-(src,dst) DISJOINT hardware, so every +/// per-direction quantity -- the pool, the counter, the coloring -- must +/// key on the ordered pair, never on the id alone. +using DirKey = std::pair; + +DirKey dirKeyOf(const Hazard &h) { + return {static_cast(h.srcPipe()), static_cast(h.dstPipe())}; +} + +} // namespace + +SyncModel mlir::pto::unified::buildSyncModel( + const SyncOperations &syncOps, unsigned bufidCapacity, + const MemInfoToClusters &memInfoToClusters, const SyncIRs &syncIR) { + SyncModel model; + model.resources.bufidCapacity = bufidCapacity; + + // --- 1. One Hazard per sync GROUP --------------------------------------- + // The outer index of SyncOperations is already the hazard: InsertSyncAnalysis + // emplaces one group per ordering, holding that ordering's set and wait (or a + // lone barrier). Taking the group as the unit is what makes the both-halves + // invariant structural rather than a convention. + for (const auto &group : llvm::enumerate(syncOps)) { + SyncOperation *setOp = nullptr; + SyncOperation *waitOp = nullptr; + for (const auto &owned : group.value()) { + SyncOperation *s = owned.get(); + if (!s || s->uselessSync) + continue; + if (s->isSyncSetType() || s->isBarrierType()) { + // A barrier group has no wait side; it occupies the `set` slot so that + // `setOp` is never null and callers need no special case to read a pipe. + if (!setOp) + setOp = s; + } else if (s->isSyncWaitType()) { + if (!waitOp) + waitOp = s; + } + } + if (!setOp) + continue; // fully-pruned group (every member uselessSync) + + Hazard h(static_cast(group.index()), setOp, waitOp); + + // --- 2. Interval, in SyncIR-index space -------------------------------- + // Forward hazard: the tight [set, wait) range. A barrier is a point. + // + // Loop-carried hazard: the CARRYING loop, not the raw [set, wait), which + // inverts because the wait is satisfied by the previous iteration. + // + // `endId + 1` is load-bearing. `Interval` is half-open, so [beginId, endId) + // would exclude position endId, which is where this hazard's own tail-wait + // sits; the colourer would then see a hazard starting at endId as disjoint + // and reuse the id while the two collide on one SyncIR element. + unsigned start = setOp->GetSyncIRIndex(); + unsigned end = waitOp ? waitOp->GetSyncIRIndex() : start; + if (setOp->GetForEndIndex().has_value()) { + const auto *loopEl = llvm::dyn_cast( + syncIR[setOp->GetForEndIndex().value()].get()); + if (loopEl) { + start = loopEl->beginId; + end = loopEl->endId + 1; + } else if (end < start) { + // GetForEndIndex did not resolve to a loop element -- keep a well-formed + // interval; synthesizeLoopCompensation will skip it and the per-kernel + // pair-count check (fe == synthesised) will surface the inconsistency. + end = start; + } + } + h.setInterval(Interval{start, end}); + + // --- 3. Join the hazard to its buffer-id cluster(s) --------------------- + // Resolved by the caller via MemAlias -- the relation BufidSyncAnalysis + // itself uses to decide two tiles are the same memory. See MemInfoToClusters + // for the three plausible-looking keys that are all wrong. + for (const BaseMemInfo *info : setOp->depMemInfos) { + if (!info) + continue; + auto it = memInfoToClusters.find(info); + if (it == memInfoToClusters.end()) + continue; + for (int c : it->second) + if (!llvm::is_contained(h.bufferClusters, c)) + h.bufferClusters.push_back(c); + } + llvm::sort(h.bufferClusters); // determinism + + model.hazards.push_back(std::move(h)); + } + + // --- 4. Is there a GENUINE reverse hazard on this buffer? ---------------- + // + // "Reverse" = a mirror-direction hazard SHARING A CLUSTER. The cluster is the + // unit because one token covers one aliasing clique. Matching on + // `depRootBuffers` -- the root allocation -- would call two hazards on different + // tiles reverse partners merely for sharing a root, and that false positive is + // the expensive direction: it prices the token low and routes a forward-only + // hazard onto it. + for (Hazard &h : model.hazards) { + if (h.isBarrier() || h.bufferClusters.empty()) + continue; + for (const Hazard &other : model.hazards) { + if (other.index() == h.index() || other.isBarrier()) + continue; + if (other.srcPipe() != h.dstPipe() || other.dstPipe() != h.srcPipe()) + continue; // not the mirror direction + for (int c : other.bufferClusters) { + if (llvm::is_contained(h.bufferClusters, c)) { + h.hasReverseWar = true; + break; + } + } + if (h.hasReverseWar) + break; + } + } + + // --- 4b. Is a token able to express this hazard at all? ------------------ + // + // Mirrors the shapes `BufidSyncCodegen` and `BufidSyncAnalysis` exclude. Kept as a + // hazard fact rather than checked at the routing site so the reason a buffer was + // refused is visible in the model report. + // + // The encodable-pipe set is the one `BufidSyncCodegen::mapPipelineToSyncOpType` + // names; a pipe outside it fails the compilation rather than degrading, so it must + // be refused here instead. + auto encodablePipe = [](PipelineType p) { + switch (p) { + case PipelineType::PIPE_MTE1: + case PipelineType::PIPE_MTE2: + case PipelineType::PIPE_MTE3: + case PipelineType::PIPE_FIX: + case PipelineType::PIPE_V: + case PipelineType::PIPE_M: + return true; + default: + return false; + } + }; + for (Hazard &h : model.hazards) { + if (h.isBarrier() || h.bufferClusters.empty()) + continue; + if (h.srcPipe() == h.dstPipe()) + continue; + if (!encodablePipe(h.srcPipe()) || !encodablePipe(h.dstPipe())) + continue; + bool scopesOk = true; + std::optional scope; + for (const BaseMemInfo *mi : h.setOp()->depMemInfos) { + if (!mi) + continue; + if (mi->scope == pto::AddressSpace::GM) { + scopesOk = false; + break; + } + if (!scope) + scope = mi->scope; + else if (*scope != mi->scope) { + scopesOk = false; + break; + } + } + if (scopesOk && scope) + h.tokenExpressible = true; + } + + // --- 4c. Address sharing between the endpoints' allocations --------------- + // + // Derived here so the model carries it with everything else it knows about a + // hazard. Read by reporting only -- see `Hazard::addressSharing`. + // + // Covers every non-barrier hazard, same-pipe included. + // + // The address is read straight off the `pto.alloc_tile` operand. `MemAlias` reaches the + // same addresses by a different route -- `BaseMemInfo::baseAddresses` guarded by + // `hasKnownPhysicalAddresses`, which PTOIRTranslator sets for a local alloc_tile whose + // addr is a non-negative constant -- so the two agree on level3. Reading the operand is + // a directness choice, not a workaround for a missing flag. + auto addrAllocOf = [](const BaseMemInfo *mi) -> pto::AllocTileOp { + if (!mi || !mi->rootBuffer) + return nullptr; + auto alloc = mi->rootBuffer.getDefiningOp(); + if (!alloc || !alloc.getAddr()) + return nullptr; + return alloc; + }; + auto addrSlotOf = [](pto::AllocTileOp alloc) + -> std::optional> { + mlir::IntegerAttr c; + if (!mlir::matchPattern(alloc.getAddr(), mlir::m_Constant(&c))) + return std::nullopt; + auto tileType = dyn_cast(alloc.getResult().getType()); + if (!tileType) + return std::nullopt; + auto asAttr = + dyn_cast_or_null(tileType.getMemorySpace()); + if (!asAttr) + return std::nullopt; + return std::make_pair(asAttr.getAddressSpace(), + static_cast(c.getInt())); + }; + for (Hazard &h : model.hazards) { + if (h.isBarrier()) + continue; + bool matched = false; + for (const BaseMemInfo *a : h.setOp()->depMemInfos) { + pto::AllocTileOp allocA = addrAllocOf(a); + if (!allocA) + continue; + auto slotA = addrSlotOf(allocA); + if (!slotA) + continue; + for (const BaseMemInfo *b : h.waitOp()->depMemInfos) { + pto::AllocTileOp allocB = addrAllocOf(b); + if (!allocB || allocB == allocA) + continue; + auto slotB = addrSlotOf(allocB); + if (!slotB || *slotA != *slotB) + continue; + + // `depMemInfos` is unordered, so which endpoint matched on which side + // means nothing. Order by program position where that is defined. + Operation *first = allocA.getOperation(); + Operation *second = allocB.getOperation(); + if (first->getBlock() == second->getBlock() && + second->isBeforeInBlock(first)) + std::swap(first, second); + + h.addressSharing = + allocA.getResult().getType() == allocB.getResult().getType() + ? Hazard::AddressSharing::Reuse + : Hazard::AddressSharing::AliasView; + h.sharedScope = slotA->first; + h.sharedAddress = slotA->second; + h.sharedAllocFirst = first; + h.sharedAllocSecond = second; + matched = true; + break; + } + if (matched) + break; + } + } + + // --- 5. Buffers -- the unit routing decides on -------------------------- + // + // Built AFTER hazards, because a buffer's facts derive from the hazards touching + // it. Assigns no id and routes nothing. + + // Loop spans, from the SyncIR. A LoopInstanceElement's [beginId, endId] is the + // span; the INNERMOST containing span is an access's loop context. + struct LoopSpanRec { + unsigned begin; + unsigned end; + }; + llvm::SmallVector loopSpans; + for (const auto &element : syncIR) { + if (auto *loop = dyn_cast_or_null(element.get())) + if (loop->getLoopKind() == pto::KindOfLoop::LOOP_BEGIN) + loopSpans.push_back({loop->beginId, loop->endId}); + } + // Innermost span containing `idx`, or -1 for function level. Innermost == the + // narrowest containing span. + auto contextOf = [&](unsigned idx) -> int { + int best = -1; + unsigned bestWidth = std::numeric_limits::max(); + for (const auto &span : llvm::enumerate(loopSpans)) { + if (idx < span.value().begin || idx > span.value().end) + continue; + unsigned width = span.value().end - span.value().begin; + if (width < bestWidth) { + bestWidth = width; + best = int(span.index()); + } + } + return best; + }; + + std::map byLogicId; + std::map> contextsOf; // logicId -> distinct loop contexts + for (const Hazard &h : model.hazards) { + if (h.isBarrier()) + continue; + for (int c : h.bufferClusters) { + Buffer &b = byLogicId[c]; + bool first = b.logicId < 0; + b.logicId = c; + // accessRange: union of the hazards' own intervals -- the tight fact. + unsigned lo = h.interval().start, hi = h.interval().end; + b.accessRange.start = first ? lo : std::min(b.accessRange.start, lo); + b.accessRange.end = first ? hi : std::max(b.accessRange.end, hi); + + // l_bufid, UNION SPAN, mirroring BufidSyncIdAlloc::computeLifeIntervals: + // an access inside a loop contributes that whole loop, one outside + // contributes itself. + for (unsigned idx : {lo, hi}) { + int ctx = contextOf(idx); + contextsOf[c].insert(ctx); + unsigned s = idx, e = idx; + if (ctx >= 0) { + s = loopSpans[ctx].begin; + e = loopSpans[ctx].end; + b.isLoopCarried = true; + } + if (!b.bufidLoopDefined) { + b.bufidLoop = {s, e}; + b.bufidLoopDefined = true; + } else { + b.bufidLoop.start = std::min(b.bufidLoop.start, s); + b.bufidLoop.end = std::max(b.bufidLoop.end, e); + } + } + } + } + + // isWrittenBack -- a fact about the BUFFER: does it carry a mirror-direction + // pair at all? Deliberately NOT `Hazard::hasReverseWar` re-read; see the + // divergence note on the struct. + for (auto &entry : byLogicId) { + int c = entry.first; + bool mirrored = false; + for (const Hazard &a : model.hazards) { + if (a.isBarrier() || !llvm::is_contained(a.bufferClusters, c)) + continue; + for (const Hazard &b2 : model.hazards) { + if (b2.isBarrier() || b2.index() == a.index()) + continue; + if (!llvm::is_contained(b2.bufferClusters, c)) + continue; + if (b2.srcPipe() == a.dstPipe() && b2.dstPipe() == a.srcPipe()) { + mirrored = true; + break; + } + } + if (mirrored) + break; + } + entry.second.isWrittenBack = mirrored; + } + + for (auto &entry : byLogicId) { + auto &b = entry.second; + const auto &ctxs = contextsOf[entry.first]; + b.loopContextCount = unsigned(ctxs.size()); + b.spansLoopBoundary = ctxs.size() > 1; + model.buffers.push_back(b); + } + + return model; +} + +//===----------------------------------------------------------------------===// +// Loop-carried compensation synthesis +//===----------------------------------------------------------------------===// + +namespace { + +// Where a synthesised head-set may sit, given the loop that carries its hazard. +// +// A head-set has no producer to follow, so it can move as early as its region +// allows. This is a cycle concern, not correctness: the consumer pipe cannot start +// until the credit is up, so a head-set parked behind unrelated compute on the +// setting pipe delays the consumer by that compute. +// +// THE BOUND IS THE CARRYING LOOP'S OWN REGION, NOT THE FUNCTION. Head and tail must +// execute the same number of times. Lifting the head out of an ENCLOSING loop while +// the tail stays inside would prime once and drain per outer iteration, so the +// second outer iteration finds no credit and the in-body wait never completes. +// +// Nothing that touches the primed buffers may be crossed, on any pipe: the credit +// permits the consumer to overwrite them, which must not be granted while a +// region-local reader or writer is still ahead of it. +unsigned hoistAnchorFor(const SyncIRs &syncIR, const LoopInstanceElement &loopEl, + const SyncOperation &headSet) { + // Region depth per element, with loop/branch markers sitting at the depth of the + // construct they delimit and bodies one deeper. + llvm::SmallVector depth(syncIR.size(), 0); + unsigned d = 0; + for (unsigned i = 0; i < syncIR.size(); ++i) { + const InstanceElement *el = syncIR[i].get(); + bool isEnd = false; + if (const auto *lp = llvm::dyn_cast(el)) + isEnd = lp->getLoopKind() == KindOfLoop::LOOP_END; + else if (const auto *br = llvm::dyn_cast(el)) + isEnd = br->getBranchKind() == KindOfBranch::IF_END; + if (isEnd && d > 0) + --d; + depth[i] = d; + bool isBegin = false; + if (const auto *lp = llvm::dyn_cast(el)) + isBegin = lp->getLoopKind() == KindOfLoop::LOOP_BEGIN; + else if (const auto *br = llvm::dyn_cast(el)) + isBegin = br->getBranchKind() == KindOfBranch::IF_BEGIN; + if (isBegin) + ++d; + } + + auto touchesPrimedBuffer = [&](const CompoundInstanceElement &cp) { + for (const BaseMemInfo *primed : headSet.depMemInfos) { + if (!primed) + continue; + for (const BaseMemInfo *other : cp.defVec) + if (other && *other == *primed) + return true; + for (const BaseMemInfo *other : cp.useVec) + if (other && *other == *primed) + return true; + } + return false; + }; + + const unsigned begin = loopEl.beginId; + if (begin >= syncIR.size()) + return begin; + const unsigned regionDepth = depth[begin]; + unsigned anchor = begin; + for (unsigned i = begin; i-- > 0;) { + if (depth[i] < regionDepth) + break; // left the carrying loop's region + const auto *cp = llvm::dyn_cast(syncIR[i].get()); + if (cp && touchesPrimedBuffer(*cp)) + break; + // Only a real op at region depth is a legal parking spot. A marker's pipeBefore + // belongs to the construct it delimits, and a deeper element is inside a sibling. + if (cp && depth[i] == regionDepth) + anchor = i; + } + return anchor; +} + +} // namespace + +unsigned mlir::pto::unified::synthesizeLoopCompensation( + SyncModel &model, SyncOperations &syncOps, SyncIRs &syncIR) { + unsigned synthesised = 0; + + // One pass, one pair per loop-carried hazard, no retry or reallocation. + for (Hazard &h : model.hazards) { + if (h.isBarrier()) + continue; + SyncOperation *inSet = h.setOp(); + if (!inSet->GetForEndIndex().has_value()) + continue; // forward hazard: its 2 ops are already sufficient + + const auto *loopEl = llvm::dyn_cast( + syncIR[inSet->GetForEndIndex().value()].get()); + if (!loopEl) + continue; // unresolved carrying loop -- the fe/synthesised count check + // in the caller reports it rather than silently priming nothing + + // The head-set clones the in-body SET at the carrying loop's begin, raising the + // flag the in-body WAIT consumes on iteration 1; the tail-wait is its + // GetMatchSync mirror at the loop's end, draining the last iteration's set. + // Both are created WITHOUT an event id -- one coloring decision stamps set, + // wait, head and tail together, and they are never coloured separately. + unsigned groupIndex = static_cast(syncOps.size()); + // `compensationOf` is what lets G2 recognise this pair as the other half of + // `inSet`'s hazard rather than a competing one. + // + // DELIBERATELY NOT `SyncOperation::isCompensation`. That flag drives codegen: + // `resolveSyncInsertAnchor` re-anchors a compensation op to its block + // terminator and can synthesise an `scf.if` else-region for it, and + // `shouldInsertBefore` forces insert-before. Nothing constructs a + // SyncOperation with `isComp=true`, so those reads are dormant and setting it + // here would activate placement logic no test covers. `compensationOf` is + // inert: only the extractor and G2 read it. + // + // The anchor is computed BEFORE the op is built, because a SyncOperation + // records its own SyncIR index and the hazard's interval must agree with it. + const unsigned headAnchor = hoistAnchorFor(syncIR, *loopEl, *inSet); + + auto headSet = std::make_unique( + inSet->GetType(), inSet->GetSrcPipe(), inSet->GetDstPipe(), groupIndex, + headAnchor, inSet->GetForEndIndex()); + headSet->compensationOf = static_cast(inSet->GetSyncIndex()); + headSet->depRootBuffers = inSet->depRootBuffers; + headSet->depMemInfos = inSet->depMemInfos; + headSet->eventIdNum = inSet->eventIdNum; + headSet->syncCoreType = inSet->syncCoreType; + headSet->SetMechanism(inSet->GetMechanism()); + + auto tailWait = headSet->GetMatchSync(loopEl->endId); + + SyncOperation *headPtr = headSet.get(); + SyncOperation *tailPtr = tailWait.get(); + + llvm::SmallVector> group; + group.emplace_back(std::move(headSet)); + group.emplace_back(std::move(tailWait)); + syncOps.emplace_back(std::move(group)); + + // --- Placement into syncIR ---------------------------------------------- + // `SyncCodegen` emits by walking syncIR's pipeBefore/pipeAfter lists, not + // `SyncOperations`, so owning the pair in `syncOps` is not enough: without + // these inserts the head/tail are dropped silently at emission -- and + // invisibly, since the syncops-view gates would still be green. Mirrors + // `SyncEventIdAllocation`'s own carrying-loop placement. + // + // THE TAIL USES push_FRONT, AND THAT IS LOAD-BEARING. The tail-wait must + // precede any existing loop-end SET at this boundary, so the loop tail anchor + // does not emit a new set before consuming the previous iteration's carried + // event. push_back yields a correct-looking pipe list that emits a broken + // sequence. + syncIR[headAnchor]->pipeBefore.push_back(headPtr); + syncIR[loopEl->endId]->pipeAfter.push_front(tailPtr); + + // The interval must COVER every op the hazard owns. Hoisting moves the head-set + // below the interval's start, so the id is held from the anchor instead; + // otherwise the colourer reuses the id while the hoisted head-set sits inside + // the other hazard's live range. Two head-sets on one element with one id are + // also indistinguishable to `MergeSyncList`, which drops the duplicate and + // leaves a wait with no set. + if (headAnchor < h.interval().start) + h.setInterval(Interval{headAnchor, h.interval().end}); + + h.setCompensation(headPtr, tailPtr); + ++synthesised; + } + + return synthesised; +} + +//===----------------------------------------------------------------------===// +// Event interval-coloring allocator +//===----------------------------------------------------------------------===// + +BufferRouteResult mlir::pto::unified::routeBuffers(SyncModel &model) { + BufferRouteResult result; + result.buffersConsidered = unsigned(model.buffers.size()); + + // Peak per-direction overlap, over ALL hazards in each direction -- the same + // quantity colorEventIds' left-edge sweep will hit. Computed by a sweep over + // interval endpoints rather than pairwise, so it is exact rather than a bound. + std::map peakOf; + std::map poolOf; + { + std::map>> events; + for (const Hazard &h : model.hazards) { + if (h.isBarrier()) + continue; + DirKey d = dirKeyOf(h); + // WEIGHTED by demand, matching the colourer: a multi-buffered hazard holds d + // ids live, not one. Unweighted, this under-predicts peak occupancy by a + // factor of d and declines to route exactly the buffers that then overflow. + int w = int(h.demand()); + events[d].push_back({h.interval().start, +w}); + events[d].push_back({h.interval().end, -w}); + poolOf[d] = model.resources.eventPoolSize(h.srcPipe(), h.dstPipe()); + } + for (auto &entry : events) { + // Sort by position; at equal positions process the releases first, because + // touching endpoints do not overlap (Interval::overlaps uses strict `<`). + llvm::sort(entry.second, [](const std::pair &a, + const std::pair &b) { + if (a.first != b.first) + return a.first < b.first; + return a.second < b.second; + }); + int live = 0; + unsigned peak = 0; + for (const auto &ev : entry.second) { + live += ev.second; + peak = std::max(peak, unsigned(std::max(live, 0))); + } + peakOf[entry.first] = peak; + } + } + + const bool haveCapacity = model.resources.bufidCapacity > 0; + + for (Buffer &b : model.buffers) { + // The directions this buffer's hazards participate in. + unsigned worstPeak = 0; + unsigned tightestPool = 0; + bool any = false; + for (const Hazard &h : model.hazards) { + if (h.isBarrier() || !llvm::is_contained(h.bufferClusters, b.logicId)) + continue; + DirKey d = dirKeyOf(h); + unsigned peak = peakOf.count(d) ? peakOf[d] : 0; + unsigned pool = poolOf.count(d) ? poolOf[d] : 0; + if (!any || peak > worstPeak) { + worstPeak = peak; + tightestPool = pool; + } + any = true; + } + b.predictedPeakOverlap = worstPeak; + b.smallestPool = tightestPool; + + if (!haveCapacity) { + ++result.skippedNoCapacity; + continue; + } + + // Second routing reason, off unless asked for: this buffer carries a hazard whose + // consuming pipe lives in only one arm of a branch its wait was hoisted out of. + // + // THE UNIT IS STILL THE BUFFER, NOT THE HAZARD, and that is forced rather than + // chosen: the token protocol is a counter cycle, so a buffer with some hazards on + // events and some on get/rls deadlocks. One qualifying hazard therefore carries + // its whole buffer across, including that buffer's other hazards. + // STRICT: omega == pool fits exactly, so routing it would be pure waste. + if (!(worstPeak > tightestPool)) { + ++result.skippedNoOverflow; + continue; + } + if (!b.isWrittenBack) { + ++result.skippedNotWrittenBack; + continue; + } + + // EVERY hazard on the buffer must be token-expressible, not just the one that + // triggered routing. Routing is all-or-nothing, so one inexpressible hazard means + // the whole buffer stays on events -- otherwise that hazard's event ops are + // suppressed and nothing replaces them. + bool allExpressible = true; + for (const Hazard &h : model.hazards) { + if (h.isBarrier() || !llvm::is_contained(h.bufferClusters, b.logicId)) + continue; + if (!h.tokenExpressible) { + allExpressible = false; + break; + } + } + if (!allExpressible) { + ++result.skippedNotExpressible; + continue; + } + b.routedToBufid = true; + ++result.routed; + } + + // ALL-OR-NOTHING. A hazard touching both a routed and an unrouted buffer cannot be + // wholly on one mechanism -- that is the split that hangs. Counted here; the caller + // refuses the whole plan when this is non-zero, before it mutates any hazard. + for (const Hazard &h : model.hazards) { + if (h.isBarrier() || h.bufferClusters.empty()) + continue; + bool anyRouted = false, anyUnrouted = false; + for (const Buffer &b : model.buffers) { + if (!llvm::is_contained(h.bufferClusters, b.logicId)) + continue; + if (b.routedToBufid) + anyRouted = true; + else + anyUnrouted = true; + } + if (anyRouted && anyUnrouted) + ++result.splitHazards; + if (anyRouted && !anyUnrouted) + ++result.hazardsCovered; + } + + return result; +} + +unsigned mlir::pto::unified::seedHiddenMacroEvents(SyncModel &model, + const SyncIRs &syncIR) { + for (size_t i = 0; i < syncIR.size(); ++i) { + auto *first = dyn_cast(syncIR[i].get()); + if (!first || first->macroOpInstanceId != 0 || !first->elementOp) + continue; + auto macro = pto::getSyncMacroModel(first->elementOp); + if (!macro || macro->hiddenEvents.empty()) + continue; + + unsigned end = first->GetIndex() + 1; + for (size_t j = i + 1; j < syncIR.size(); ++j) { + auto *other = dyn_cast(syncIR[j].get()); + if (!other || other->elementOp != first->elementOp) + continue; + end = other->GetIndex(); + } + unsigned begin = first->GetIndex(); + if (begin > 0) + --begin; + if (end + 1 < syncIR.size()) + ++end; + if (begin >= end) + continue; + + for (const auto &hidden : macro->hiddenEvents) + for (unsigned id : hidden.eventIds) + model.hiddenReservations.push_back( + {static_cast(hidden.srcPipe), static_cast(hidden.dstPipe), + static_cast(id), Interval{begin, end}, + first->elementOp->getName().getStringRef()}); + } + return model.hiddenReservations.size(); +} + +EventColorResult mlir::pto::unified::colorEventIds(SyncModel &model) { + EventColorResult result; + + // Non-barrier hazards, ordered for a per-direction left-edge scan. The order + // is (direction, interval start, hazard index) -- fully deterministic, and it + // groups each direction into a contiguous run below. + llvm::SmallVector haz; + for (Hazard &h : model.hazards) { + if (h.isBarrier()) + continue; + // Routed to the token protocol, so it must NOT take an event id -- a buffer + // with some hazards on events and some on get/rls is a counter cycle missing a + // step, i.e. a hang. Skipped, not counted as overflow: overflow means no id was + // available, this means none was wanted. + if (h.mechanism() == SyncOperation::MECHANISM::BUFID) { + ++result.skippedRouted; + continue; + } + haz.push_back(&h); + } + llvm::stable_sort(haz, [](const Hazard *a, const Hazard *b) { + DirKey da = dirKeyOf(*a), db = dirKeyOf(*b); + if (da != db) + return da < db; + if (a->interval().start != b->interval().start) + return a->interval().start < b->interval().start; + return a->index() < b->index(); + }); + + // Color one direction at a time; each is an independent interval graph, so a + // left-edge sweep with the lowest free id attains its chromatic number. + for (size_t i = 0; i < haz.size();) { + DirKey dir = dirKeyOf(*haz[i]); + unsigned pool = + model.resources.eventPoolSize(haz[i]->srcPipe(), haz[i]->dstPipe()); + llvm::SmallVector endOf(pool, 0); + llvm::SmallVector inUse(pool, false); + llvm::SmallVector everUsed(pool, false); + + for (; i < haz.size() && dirKeyOf(*haz[i]) == dir; ++i) { + Hazard *h = haz[i]; + unsigned start = h->interval().start; + + // Left-edge release: an id whose interval ended at or before this start is + // free again. Touching endpoints do not overlap (Interval::overlaps uses + // strict `<`), so the test is `<=`. + for (unsigned id = 0; id < pool; ++id) + if (inUse[id] && endOf[id] <= start) + inUse[id] = false; + + // The d LOWEST free ids: optimal weighted coloring, deterministic. d == 1 for + // an ordinary hazard, > 1 only for a multi-buffered pair. + // + // ALL-OR-NOTHING: fewer than d ids is not a degraded assignment but an + // out-of-bounds read, since `CreateSetWaitOpForMultiBuffer` indexes + // eventIds[i] for i in [0, slotCount). A short pool falls through to the + // spill. + unsigned d = h->demand(); + // An id is unavailable if a macro's library implementation holds it over a + // span overlapping this hazard. Checked per hazard rather than seeded into + // `inUse`, because a reservation is an INTERVAL: seeding would also block + // hazards that start before the call, which is conservative but wrong. + auto reservedFor = [&](unsigned id) { + for (const HiddenReservation &r : model.hiddenReservations) + if (r.id == static_cast(id) && r.srcPipe == dir.first && + r.dstPipe == dir.second && r.span.overlaps(h->interval())) + return true; + return false; + }; + // DO NOT HAND BACK THE MOST RECENTLY FREED ID while another is free. An event + // flag carries one outstanding signal, so two hazards sharing an id cannot + // both be in flight and the producing pipe can never run more than one signal + // ahead. Alternating between two ids restores that one step of run-ahead; + // a third buys no further overlap, so this avoids one id rather than cycling + // the pool. + // + // Peak usage stays bounded by min(pool, omega + 1): the avoidance applies only + // when a second id is free, so it cannot turn a fitting assignment into a + // spill. + int avoid = -1; + unsigned avoidEnd = 0; + for (unsigned id = 0; id < pool; ++id) + if (!inUse[id] && everUsed[id] && !reservedFor(id) && + (avoid < 0 || endOf[id] > avoidEnd)) { + avoidEnd = endOf[id]; + avoid = static_cast(id); + } + + auto collect = [&](int skip) { + llvm::SmallVector out; + for (unsigned id = 0; id < pool && out.size() < d; ++id) + if (!inUse[id] && !reservedFor(id) && static_cast(id) != skip) + out.push_back(static_cast(id)); + return out; + }; + llvm::SmallVector chosenIds = collect(avoid); + if (chosenIds.size() < d) + chosenIds = collect(-1); // not enough without it; the spread is a preference + int chosen = chosenIds.size() == d ? chosenIds.front() : -1; + if (chosen < 0) { + // No id free at this instant in this direction. SPILL TO A BARRIER. + // "No id" is not a resting state: codegen emits the pair regardless, so an + // unserved hazard goes out carrying id 0 or the unset sentinel. + // + // There is no "spill failed" branch below, so the assumption that the + // barrier class is always available is asserted where it is relied on: a + // bounded barrier class cannot be introduced without this failing to + // compile until a failure mode is written here. + static_assert(ResourceModel::barrierIsUnbounded, + "the spill path has no failure branch, so the barrier class " + "must be unbounded and always available"); + ++result.overflow; + ++result.spilledToBarrier; + h->SpillToBarrier(); + continue; + } + + for (int id : chosenIds) { + inUse[id] = true; + endOf[id] = h->interval().end; + if (everUsed[static_cast(id)]) + ++result.reused; + everUsed[static_cast(id)] = true; + } + result.idsAssigned += d; + if (d > 1) + ++result.rotating; + + // Backward hazards need no special case: compensation synthesis already gave + // each a head/tail pair and a forward carrying-loop interval, so they colour + // like any other. Stamps set, wait, head and tail with the SAME id list -- + // a compensation pair left unstamped emits with the wrong arity. + h->SetEventIds(chosenIds); + ++result.assigned; + } + } + return result; +} + +//===----------------------------------------------------------------------===// +// Rendering +//===----------------------------------------------------------------------===// + +void mlir::pto::unified::printSyncModel(llvm::raw_ostream &os, + llvm::StringRef funcName, + const SyncModel &model) { + os << "[unified-model] func=" << funcName + << " hazards=" << model.hazards.size() + << " K=" << model.resources.bufidCapacity << "\n"; + + for (const Hazard &h : model.hazards) { + Alpha a = Alpha::forHazard(h); + os << " h#" << h.index() << " " + << (h.isBarrier() ? "barrier" : "pair") + << " " << oracle::pipelineTypeName(h.srcPipe()) << "->" + << oracle::pipelineTypeName(h.dstPipe()) + << " iv=[" << h.interval().start << "," << h.interval().end << ")" + << " pool=" << model.resources.eventPoolSize(h.srcPipe(), h.dstPipe()) + << " mech=" << SyncOperation::MechanismName(h.mechanism()) + << " clusters=["; + for (size_t i = 0; i < h.bufferClusters.size(); ++i) { + if (i) + os << ","; + os << h.bufferClusters[i]; + } + os << "]" + << " revwar=" << (h.hasReverseWar ? "yes" : "no") + << " share=" + << (h.addressSharing == Hazard::AddressSharing::Reuse + ? "reuse" + : h.addressSharing == Hazard::AddressSharing::AliasView + ? "alias" + : "none") + << " alpha(e/b/bar)=" << a.event << "/" << a.bufid << "/" << a.barrier + << " event_id="; + // ALL ids, not just the first: a rotating hazard holds d of them. + if (h.hasEventId()) { + const auto &ids = h.setOp()->eventIds; + for (size_t i = 0; i < ids.size(); ++i) { + if (i) + os << "/"; + os << ids[i]; + } + } else { + os << "-"; + } + os << "\n"; + } + + // `lbufid=D/N` says how many of the N buffers got a resolved reservation window; + // D != N is a real gap, not a default. `span_boundary` counts buffers with no + // single carrying loop, which are given a union span rather than refused. + // `revwar_divergence` counts buffers where the per-BUFFER `isWrittenBack` + // disagrees with a hazard's per-HAZARD `hasReverseWar`, so the two predicates + // cannot silently stand in for one another. + if (!model.buffers.empty()) { + unsigned wb = 0, lc = 0, sb = 0, defined = 0, divergence = 0; + for (const Buffer &b : model.buffers) { + if (b.isWrittenBack) + ++wb; + if (b.isLoopCarried) + ++lc; + if (b.spansLoopBoundary) + ++sb; + if (b.bufidLoopDefined) + ++defined; + for (const Hazard &h : model.hazards) { + if (h.isBarrier() || !llvm::is_contained(h.bufferClusters, b.logicId)) + continue; + if (h.hasReverseWar != b.isWrittenBack) { + ++divergence; + break; + } + } + } + os << " buffers=" << model.buffers.size() << " written_back=" << wb + << " loop_carried=" << lc << " span_boundary=" << sb + << " lbufid=" << defined << "/" << model.buffers.size() + << " revwar_divergence=" << divergence << "\n"; + for (const Buffer &b : model.buffers) + os << " b#" << b.logicId << " access=[" << b.accessRange.start << "," + << b.accessRange.end << ") lbufid=" + << (b.bufidLoopDefined ? "[" + std::to_string(b.bufidLoop.start) + "," + + std::to_string(b.bufidLoop.end) + "]" + : std::string("UNRESOLVED")) + << " wb=" << (b.isWrittenBack ? "yes" : "no") + << " loop=" << (b.isLoopCarried ? "yes" : "no") + << " ctx=" << b.loopContextCount + << (b.spansLoopBoundary ? " SPANS-BOUNDARY" : "") << "\n"; + } +} diff --git a/test/lit/pto/Inputs/addr_reuse_war.pto b/test/lit/pto/Inputs/addr_reuse_war.pto new file mode 100644 index 0000000000..b9100b57f9 --- /dev/null +++ b/test/lit/pto/Inputs/addr_reuse_war.pto @@ -0,0 +1,55 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// Shared input for the address-reuse diagnostic. Two allocation pairs share an +// address, differing in exactly what the diagnostic classifies on: +// addr 0 -- two tiles of the SAME type, the second reusing the first's space +// addr 512 -- two tiles of DIFFERENT type over the same bytes, an alias view +// +// The two orderings are given DIFFERENT pipe directions on purpose. With the +// same direction, program order lets one sync cover both and RemoveRedundantSync +// keeps only one, leaving the other classification untested. +// +// Explicit `addr` operands are accepted only under --pto-level=level3. +// lit excludes this directory from discovery. + +module { + func.func @addr_reuse_war(%a: !pto.partition_tensor_view<1x64xf32>, + %b: !pto.partition_tensor_view<1x64xf32>, + %c: !pto.partition_tensor_view<1x64xf32>, + %d: !pto.partition_tensor_view<8x8xf32>) { + %a0 = arith.constant 0 : i64 + %a512 = arith.constant 512 : i64 + + %t0 = pto.alloc_tile addr = %a0 : !pto.tile_buf + %t1 = pto.alloc_tile addr = %a0 : !pto.tile_buf + %v0 = pto.alloc_tile addr = %a512 : !pto.tile_buf + %v1 = pto.alloc_tile addr = %a512 : !pto.tile_buf + + pto.tload ins(%a : !pto.partition_tensor_view<1x64xf32>) + outs(%t0 : !pto.tile_buf) + pto.tload ins(%b : !pto.partition_tensor_view<1x64xf32>) + outs(%v0 : !pto.tile_buf) + + // V reads %v0; the load below writes %v1 over the same bytes: V -> MTE2. + pto.tadd ins(%t0, %v0 : !pto.tile_buf, + !pto.tile_buf) + outs(%t0 : !pto.tile_buf) + pto.tload ins(%d : !pto.partition_tensor_view<8x8xf32>) + outs(%v1 : !pto.tile_buf) + + // MTE3 reads %t0; the load below writes %t1 over the same bytes: MTE3 -> MTE2. + pto.tstore ins(%t0 : !pto.tile_buf) + outs(%c : !pto.partition_tensor_view<1x64xf32>) + pto.tload ins(%a : !pto.partition_tensor_view<1x64xf32>) + outs(%t1 : !pto.tile_buf) + pto.tstore ins(%t1 : !pto.tile_buf) + outs(%c : !pto.partition_tensor_view<1x64xf32>) + return + } +} diff --git a/test/lit/pto/Inputs/coverage_profile_v1.cov b/test/lit/pto/Inputs/coverage_profile_v1.cov new file mode 100644 index 0000000000..b8706c3437 --- /dev/null +++ b/test/lit/pto/Inputs/coverage_profile_v1.cov @@ -0,0 +1 @@ +[sync-cov] func=oracle_vadd anchors=4 sig=0x123456789abcdef0 edge_count=3 edges=0-2,1-2,2-3 bedge_count=0 bedges= barrier_all=1 diff --git a/test/lit/pto/Inputs/oracle_buf_token_spellings.pto b/test/lit/pto/Inputs/oracle_buf_token_spellings.pto new file mode 100644 index 0000000000..ae0894372c --- /dev/null +++ b/test/lit/pto/Inputs/oracle_buf_token_spellings.pto @@ -0,0 +1,41 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// One buffer-token chain, written twice in two of the spellings `PTO_PipeLikeAttr` +// admits. Both name the same two pipes and establish the same ordering, so a gate +// that decodes only one spelling reports different coverage for identical programs. +// lit excludes this directory from discovery. + +module { + // The plain PipeAttr spelling. + func.func @buf_chain_pipe_attr(%a: !pto.partition_tensor_view<1x64xf32>, + %c: !pto.partition_tensor_view<1x64xf32>) { + %t = pto.alloc_tile : !pto.tile_buf + pto.get_buf "PIPE_MTE2", 0, 0 + pto.tload ins(%a : !pto.partition_tensor_view<1x64xf32>) outs(%t : !pto.tile_buf) + pto.rls_buf "PIPE_MTE2", 0, 0 + pto.get_buf "PIPE_MTE3", 0, 0 + pto.tstore ins(%t : !pto.tile_buf) outs(%c : !pto.partition_tensor_view<1x64xf32>) + pto.rls_buf "PIPE_MTE3", 0, 0 + return + } + + // The pipe_event_type spelling. TLOAD maps to MTE2 and TSTORE_VEC to MTE3, so this + // is the same chain over the same pipes as above. + func.func @buf_chain_event_type(%a: !pto.partition_tensor_view<1x64xf32>, + %c: !pto.partition_tensor_view<1x64xf32>) { + %t = pto.alloc_tile : !pto.tile_buf + pto.get_buf[#pto.pipe_event_type, 0] + pto.tload ins(%a : !pto.partition_tensor_view<1x64xf32>) outs(%t : !pto.tile_buf) + pto.rls_buf[#pto.pipe_event_type, 0] + pto.get_buf[#pto.pipe_event_type, 0] + pto.tstore ins(%t : !pto.tile_buf) outs(%c : !pto.partition_tensor_view<1x64xf32>) + pto.rls_buf[#pto.pipe_event_type, 0] + return + } +} diff --git a/test/lit/pto/Inputs/oracle_event_pair_guards.pto b/test/lit/pto/Inputs/oracle_event_pair_guards.pto new file mode 100644 index 0000000000..381a4e3285 --- /dev/null +++ b/test/lit/pto/Inputs/oracle_event_pair_guards.pto @@ -0,0 +1,98 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// One event id, placed five ways. Three of them leave a runtime path that executes +// one half of the pair without the other; two are legal and must stay silent. +// lit excludes this directory from discovery. + +module { + // The set on one arm, the wait on the other. No path runs both, yet the counts + // balance and a flattened scan pops this wait against this set. + func.func @pair_split_across_arms(%a: !pto.partition_tensor_view<1x64xf32>, + %c: !pto.partition_tensor_view<1x64xf32>, %p: i1) { + pto.section.vector { + %t = pto.alloc_tile : !pto.tile_buf + pto.tload ins(%a : !pto.partition_tensor_view<1x64xf32>) outs(%t : !pto.tile_buf) + scf.if %p { + pto.set_flag[, , ] + } else { + pto.wait_flag[, , ] + } + pto.tstore ins(%t : !pto.tile_buf) outs(%c : !pto.partition_tensor_view<1x64xf32>) + } + return + } + + // A guarded set with an unguarded wait: when the guard is false the wait has + // nothing to consume and the kernel hangs. + func.func @guarded_set_plain_wait(%a: !pto.partition_tensor_view<1x64xf32>, + %c: !pto.partition_tensor_view<1x64xf32>, %p: i1) { + pto.section.vector { + %t = pto.alloc_tile : !pto.tile_buf + pto.tload ins(%a : !pto.partition_tensor_view<1x64xf32>) outs(%t : !pto.tile_buf) + scf.if %p { + pto.set_flag[, , ] + } + pto.wait_flag[, , ] + pto.tstore ins(%t : !pto.tile_buf) outs(%c : !pto.partition_tensor_view<1x64xf32>) + } + return + } + + // The mirror: an unguarded set with a guarded wait leaks the id on the false path. + func.func @plain_set_guarded_wait(%a: !pto.partition_tensor_view<1x64xf32>, + %c: !pto.partition_tensor_view<1x64xf32>, %p: i1) { + pto.section.vector { + %t = pto.alloc_tile : !pto.tile_buf + pto.tload ins(%a : !pto.partition_tensor_view<1x64xf32>) outs(%t : !pto.tile_buf) + pto.set_flag[, , ] + scf.if %p { + pto.wait_flag[, , ] + } + pto.tstore ins(%t : !pto.tile_buf) outs(%c : !pto.partition_tensor_view<1x64xf32>) + } + return + } + + // LEGAL. Both halves on the same arm: the pair is equally guarded, so either the + // whole pair runs or neither half does. + func.func @pair_on_one_arm(%a: !pto.partition_tensor_view<1x64xf32>, + %c: !pto.partition_tensor_view<1x64xf32>, %p: i1) { + pto.section.vector { + %t = pto.alloc_tile : !pto.tile_buf + pto.tload ins(%a : !pto.partition_tensor_view<1x64xf32>) outs(%t : !pto.tile_buf) + scf.if %p { + pto.set_flag[, , ] + pto.wait_flag[, , ] + } + pto.tstore ins(%t : !pto.tile_buf) outs(%c : !pto.partition_tensor_view<1x64xf32>) + } + return + } + + // LEGAL, AND THE SHAPE THAT BOUNDS THE RULE. The set is outside the loop and the + // wait inside it, so the two sit in different BLOCKS while carrying the same + // conditional guards. This is what the emitter overwhelmingly produces, and a rule + // comparing blocks rather than conditionals would reject it. + func.func @set_outside_loop_wait_inside(%a: !pto.partition_tensor_view<1x64xf32>, + %c: !pto.partition_tensor_view<1x64xf32>) { + pto.section.vector { + %c0 = arith.constant 0 : index + %c1 = arith.constant 1 : index + %c4 = arith.constant 4 : index + %t = pto.alloc_tile : !pto.tile_buf + pto.tload ins(%a : !pto.partition_tensor_view<1x64xf32>) outs(%t : !pto.tile_buf) + pto.set_flag[, , ] + scf.for %i = %c0 to %c4 step %c1 { + pto.wait_flag[, , ] + } + pto.tstore ins(%t : !pto.tile_buf) outs(%c : !pto.partition_tensor_view<1x64xf32>) + } + return + } +} diff --git a/test/lit/pto/Inputs/oracle_gate_features.pto b/test/lit/pto/Inputs/oracle_gate_features.pto new file mode 100644 index 0000000000..2b90f3aeb3 --- /dev/null +++ b/test/lit/pto/Inputs/oracle_gate_features.pto @@ -0,0 +1,44 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// Two shapes G1-self treats specially, isolated so each can be pinned on its own. +// lit excludes this directory from discovery. + +module { + // Two consecutive PIPE_V ops on one tile: a same-pipe read-after-write. A5 + // guarantees intra-pipe ordering on that pipe and its backend rejects an + // explicit vector barrier, so the gate must credit the ordering there and must + // NOT credit it on A3. + func.func @same_pipe(%a: !pto.partition_tensor_view<1x64xf32>, + %c: !pto.partition_tensor_view<1x64xf32>) { + %t = pto.alloc_tile : !pto.tile_buf + %u = pto.alloc_tile : !pto.tile_buf + pto.tload ins(%a : !pto.partition_tensor_view<1x64xf32>) outs(%t : !pto.tile_buf) + pto.tadd ins(%t, %t : !pto.tile_buf, !pto.tile_buf) outs(%u : !pto.tile_buf) + pto.tmul ins(%u, %u : !pto.tile_buf, !pto.tile_buf) outs(%u : !pto.tile_buf) + pto.tstore ins(%u : !pto.tile_buf) outs(%c : !pto.partition_tensor_view<1x64xf32>) + return + } + + // The two arms of one `scf.if` can never both execute, so no ordering is owed + // between an op in each. Counted rather than dropped, so the bookkeeping identity + // still balances and a predicate that swallowed real dependencies would show up. + func.func @exclusive_arms(%a: !pto.partition_tensor_view<1x64xf32>, + %c: !pto.partition_tensor_view<1x64xf32>, %p: i1) { + %t = pto.alloc_tile : !pto.tile_buf + %u = pto.alloc_tile : !pto.tile_buf + pto.tload ins(%a : !pto.partition_tensor_view<1x64xf32>) outs(%t : !pto.tile_buf) + scf.if %p { + pto.tadd ins(%t, %t : !pto.tile_buf, !pto.tile_buf) outs(%u : !pto.tile_buf) + } else { + pto.tmul ins(%t, %t : !pto.tile_buf, !pto.tile_buf) outs(%u : !pto.tile_buf) + } + pto.tstore ins(%u : !pto.tile_buf) outs(%c : !pto.partition_tensor_view<1x64xf32>) + return + } +} diff --git a/test/lit/pto/Inputs/oracle_selfval_illegal_id.pto b/test/lit/pto/Inputs/oracle_selfval_illegal_id.pto new file mode 100644 index 0000000000..3d046b5ce4 --- /dev/null +++ b/test/lit/pto/Inputs/oracle_selfval_illegal_id.pto @@ -0,0 +1,23 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// Oracle self-validation fixture: ONE fault, of G3's class only. +// V->S reserves the top id of its pool, so EVENT_ID7 is illegal there. The +// set/wait interval discipline is perfect, so G2 must stay silent. +// Event-only on purpose: buffer-id sync is A5-only and this fixture is compiled for +// a3, where the exposed capacity is zero, so a get_buf here would be rejected for the +// arch rather than for the id under test. +// Not a test: lives under Inputs/, which lit excludes from discovery. + +module attributes {pto.target_arch = "a3"} { + func.func @selfval_illegal_id() attributes {pto.kernel_kind = #pto.kernel_kind} { + pto.set_flag["PIPE_V", "PIPE_S", "EVENT_ID7"] + pto.wait_flag["PIPE_V", "PIPE_S", "EVENT_ID7"] + return + } +} diff --git a/test/lit/pto/Inputs/oracle_selfval_nested_id.pto b/test/lit/pto/Inputs/oracle_selfval_nested_id.pto new file mode 100644 index 0000000000..e5af38a0b9 --- /dev/null +++ b/test/lit/pto/Inputs/oracle_selfval_nested_id.pto @@ -0,0 +1,23 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// Oracle self-validation fixture: ONE fault, of G2's class only. +// Event id 0 on MTE2->V is opened twice before either close: two overlapping +// hazards share it. Every id is inside its pool and its direction reserves +// nothing, so G3 must stay silent. +// Not a test: lives under Inputs/, which lit excludes from discovery. + +module attributes {pto.target_arch = "a3"} { + func.func @selfval_nested_id() attributes {pto.kernel_kind = #pto.kernel_kind} { + pto.set_flag["PIPE_MTE2", "PIPE_V", "EVENT_ID0"] + pto.set_flag["PIPE_MTE2", "PIPE_V", "EVENT_ID0"] + pto.wait_flag["PIPE_MTE2", "PIPE_V", "EVENT_ID0"] + pto.wait_flag["PIPE_MTE2", "PIPE_V", "EVENT_ID0"] + return + } +} diff --git a/test/lit/pto/Inputs/oracle_sync_reachability.pto b/test/lit/pto/Inputs/oracle_sync_reachability.pto new file mode 100644 index 0000000000..ffde1f3ba5 --- /dev/null +++ b/test/lit/pto/Inputs/oracle_sync_reachability.pto @@ -0,0 +1,90 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// Four placements of one barrier, distinguishing "the sync sits before the sink in +// program order" from "the sync is reached on every path that reaches the sink". +// lit excludes this directory from discovery. + +module { + // The barrier runs only on the `then` path; both stores run regardless. On the + // false path they execute back to back with nothing ordering them. + func.func @sync_in_arm_outside_sink(%a: !pto.partition_tensor_view<1x64xf32>, + %c: !pto.partition_tensor_view<1x64xf32>, %p: i1) { + pto.section.vector { + %t = pto.alloc_tile : !pto.tile_buf + pto.tload ins(%a : !pto.partition_tensor_view<1x64xf32>) outs(%t : !pto.tile_buf) + pto.set_flag[, , ] + pto.wait_flag[, , ] + pto.tstore ins(%t : !pto.tile_buf) outs(%c : !pto.partition_tensor_view<1x64xf32>) + scf.if %p { + pto.barrier + } + pto.tstore ins(%t : !pto.tile_buf) outs(%c : !pto.partition_tensor_view<1x64xf32>) + } + return + } + + // The same kernel with the barrier hoisted out of the `if`. Identical dependencies, + // and this one really is ordered. + func.func @sync_hoisted_covers(%a: !pto.partition_tensor_view<1x64xf32>, + %c: !pto.partition_tensor_view<1x64xf32>, %p: i1) { + pto.section.vector { + %t = pto.alloc_tile : !pto.tile_buf + pto.tload ins(%a : !pto.partition_tensor_view<1x64xf32>) outs(%t : !pto.tile_buf) + pto.set_flag[, , ] + pto.wait_flag[, , ] + pto.tstore ins(%t : !pto.tile_buf) outs(%c : !pto.partition_tensor_view<1x64xf32>) + pto.barrier + pto.tstore ins(%t : !pto.tile_buf) outs(%c : !pto.partition_tensor_view<1x64xf32>) + } + return + } + + // LEGAL. Barrier and sink in the SAME arm: every path reaching the sink passes + // through the barrier, so the ordering holds and the edge must survive. Rejecting + // this is what a rule keyed on the source side as well would do. + func.func @sync_and_sink_in_one_arm(%a: !pto.partition_tensor_view<1x64xf32>, + %c: !pto.partition_tensor_view<1x64xf32>, %p: i1) { + pto.section.vector { + %t = pto.alloc_tile : !pto.tile_buf + pto.tload ins(%a : !pto.partition_tensor_view<1x64xf32>) outs(%t : !pto.tile_buf) + pto.set_flag[, , ] + pto.wait_flag[, , ] + pto.tstore ins(%t : !pto.tile_buf) outs(%c : !pto.partition_tensor_view<1x64xf32>) + scf.if %p { + pto.barrier + pto.tstore ins(%t : !pto.tile_buf) outs(%c : !pto.partition_tensor_view<1x64xf32>) + } + } + return + } + + // LEGAL, AND THE BOUND ON THE RULE. The barrier is inside a loop whose trip count is + // a constant 3, and the sink follows the loop. The barrier therefore always runs and + // the ordering holds -- but STRUCTURAL DOMINANCE says a body op never dominates an + // op after the loop, so a dominance-keyed rule reports this as uncovered. That is + // the false positive this line exists to catch. + func.func @sync_in_loop_outside_sink(%a: !pto.partition_tensor_view<1x64xf32>, + %c: !pto.partition_tensor_view<1x64xf32>) { + pto.section.vector { + %c0 = arith.constant 0 : index + %c1 = arith.constant 1 : index + %c3 = arith.constant 3 : index + %t = pto.alloc_tile : !pto.tile_buf + pto.tload ins(%a : !pto.partition_tensor_view<1x64xf32>) outs(%t : !pto.tile_buf) + pto.set_flag[, , ] + pto.wait_flag[, , ] + scf.for %i = %c0 to %c3 step %c1 { + pto.tstore ins(%t : !pto.tile_buf) outs(%c : !pto.partition_tensor_view<1x64xf32>) + pto.barrier + } + pto.tstore ins(%t : !pto.tile_buf) outs(%c : !pto.partition_tensor_view<1x64xf32>) + } + return + } +} diff --git a/test/lit/pto/Inputs/oracle_vadd.pto b/test/lit/pto/Inputs/oracle_vadd.pto new file mode 100644 index 0000000000..7682b4f5df --- /dev/null +++ b/test/lit/pto/Inputs/oracle_vadd.pto @@ -0,0 +1,31 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// Shared input for the oracle gate tests: two loads, one vector add, one store. +// That is the smallest kernel carrying both a cross-pipe read-after-write +// (MTE2 -> V) and a write-after-read the other way (V -> MTE3), so every gate has +// something real to judge. lit excludes this directory from discovery. + +module { + func.func @oracle_vadd(%a: !pto.partition_tensor_view<1x64xf32>, + %b: !pto.partition_tensor_view<1x64xf32>, + %c: !pto.partition_tensor_view<1x64xf32>) { + %ta = pto.alloc_tile : !pto.tile_buf + %tb = pto.alloc_tile : !pto.tile_buf + pto.tload ins(%a : !pto.partition_tensor_view<1x64xf32>) + outs(%ta : !pto.tile_buf) + pto.tload ins(%b : !pto.partition_tensor_view<1x64xf32>) + outs(%tb : !pto.tile_buf) + pto.tadd ins(%ta, %tb : !pto.tile_buf, + !pto.tile_buf) + outs(%ta : !pto.tile_buf) + pto.tstore ins(%ta : !pto.tile_buf) + outs(%c : !pto.partition_tensor_view<1x64xf32>) + return + } +} diff --git a/test/lit/pto/Inputs/routed_token_ids.pto b/test/lit/pto/Inputs/routed_token_ids.pto new file mode 100644 index 0000000000..b1bc8c1580 --- /dev/null +++ b/test/lit/pto/Inputs/routed_token_ids.pto @@ -0,0 +1,81 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// The routing kernel from sync_forced_overflow, alone in a file so that absence checks +// cover the whole output rather than colliding with the unrouted kernels beside it. +// Twelve tiles overflow the MTE2->V event pool and every one routes to a buffer-id token; +// the accumulator does not overflow and stays on events. +// lit excludes this directory from discovery. + +module { + func.func @overflow_writeback(%src: !pto.ptr, %dst: !pto.ptr) attributes {pto.kernel_kind = #pto.kernel_kind} { + %c0 = arith.constant 0 : index + %c1 = arith.constant 1 : index + %c32 = arith.constant 32 : index + %c4 = arith.constant 4 : index + %a0 = arith.constant 0 : i64 + %a1 = arith.constant 2048 : i64 + %a2 = arith.constant 4096 : i64 + %a3 = arith.constant 6144 : i64 + %a4 = arith.constant 8192 : i64 + %a5 = arith.constant 10240 : i64 + %a6 = arith.constant 12288 : i64 + %a7 = arith.constant 14336 : i64 + %a8 = arith.constant 16384 : i64 + %a9 = arith.constant 18432 : i64 + %a10 = arith.constant 20480 : i64 + %a11 = arith.constant 22528 : i64 + %aout = arith.constant 24576 : i64 + %sv = pto.make_tensor_view %src, shape = [%c32, %c32], strides = [%c32, %c1] : !pto.tensor_view + %dv = pto.make_tensor_view %dst, shape = [%c32, %c32], strides = [%c32, %c1] : !pto.tensor_view + %ps = pto.partition_view %sv, offsets = [%c0, %c0], sizes = [%c32, %c32] : !pto.tensor_view -> !pto.partition_tensor_view<32x32xf16> + %pd = pto.partition_view %dv, offsets = [%c0, %c0], sizes = [%c32, %c32] : !pto.tensor_view -> !pto.partition_tensor_view<32x32xf16> + scf.for %it = %c0 to %c4 step %c1 { + %t0 = pto.alloc_tile addr = %a0 : !pto.tile_buf + %t1 = pto.alloc_tile addr = %a1 : !pto.tile_buf + %t2 = pto.alloc_tile addr = %a2 : !pto.tile_buf + %t3 = pto.alloc_tile addr = %a3 : !pto.tile_buf + %t4 = pto.alloc_tile addr = %a4 : !pto.tile_buf + %t5 = pto.alloc_tile addr = %a5 : !pto.tile_buf + %t6 = pto.alloc_tile addr = %a6 : !pto.tile_buf + %t7 = pto.alloc_tile addr = %a7 : !pto.tile_buf + %t8 = pto.alloc_tile addr = %a8 : !pto.tile_buf + %t9 = pto.alloc_tile addr = %a9 : !pto.tile_buf + %t10 = pto.alloc_tile addr = %a10 : !pto.tile_buf + %t11 = pto.alloc_tile addr = %a11 : !pto.tile_buf + %tout = pto.alloc_tile addr = %aout : !pto.tile_buf + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t0 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t1 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t2 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t3 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t4 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t5 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t6 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t7 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t8 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t9 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t10 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t11 : !pto.tile_buf) + pto.tcvt ins(%t0{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t1{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t2{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t3{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t4{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t5{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t6{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t7{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t8{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t9{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t10{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t11{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tstore ins(%tout : !pto.tile_buf) outs(%pd : !pto.partition_tensor_view<32x32xf16>) + scf.yield + } + return + } +} diff --git a/test/lit/pto/Inputs/unified_model_loop_carried.pto b/test/lit/pto/Inputs/unified_model_loop_carried.pto new file mode 100644 index 0000000000..48dca08852 --- /dev/null +++ b/test/lit/pto/Inputs/unified_model_loop_carried.pto @@ -0,0 +1,59 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// Shared input: four tiles, each loaded, computed and stored inside one loop, at +// level3 so the addresses are explicit. Every tile's store is followed by the next +// iteration's load of the same tile, so each contributes one LOOP-CARRIED hazard -- +// four in total, which is what makes the compensation counters discriminate: a +// synthesiser that primed one pair, or one per loop rather than one per hazard, +// would disagree with `fe`. The four disjoint addresses also give the colourer +// something to reuse ids across. lit excludes this directory from discovery. + +module attributes {pto.target_arch = "a2a3"} { + func.func @loop_carried_tiles(%src: !pto.ptr, %dst: !pto.ptr, %n: index) + attributes {pto.kernel_kind = #pto.kernel_kind} { + %c0_i64 = arith.constant 0 : i64 + %a1 = arith.constant 512 : i64 + %a2 = arith.constant 1024 : i64 + %a3 = arith.constant 1536 : i64 + %c0 = arith.constant 0 : index + %c1 = arith.constant 1 : index + %c16 = arith.constant 16 : index + %c64 = arith.constant 64 : index + + %sv = pto.make_tensor_view %src, shape = [%c64, %c64], strides = [%c64, %c1] {layout = #pto.layout} : !pto.tensor_view + %dv = pto.make_tensor_view %dst, shape = [%c64, %c64], strides = [%c64, %c1] {layout = #pto.layout} : !pto.tensor_view + + %t0 = pto.alloc_tile addr = %c0_i64 valid_row = %c16 valid_col = %c16 : !pto.tile_buf + %t1 = pto.alloc_tile addr = %a1 valid_row = %c16 valid_col = %c16 : !pto.tile_buf + %t2 = pto.alloc_tile addr = %a2 valid_row = %c16 valid_col = %c16 : !pto.tile_buf + %t3 = pto.alloc_tile addr = %a3 valid_row = %c16 valid_col = %c16 : !pto.tile_buf + + scf.for %i = %c0 to %n step %c1 { + %sp = pto.partition_view %sv, offsets = [%c0, %c0], sizes = [%c16, %c16] : !pto.tensor_view -> !pto.partition_tensor_view<16x16xf16> + %dp = pto.partition_view %dv, offsets = [%c0, %c0], sizes = [%c16, %c16] : !pto.tensor_view -> !pto.partition_tensor_view<16x16xf16> + + pto.tload ins(%sp : !pto.partition_tensor_view<16x16xf16>) outs(%t0 : !pto.tile_buf) + pto.tadd ins(%t0, %t0 : !pto.tile_buf, !pto.tile_buf) outs(%t0 : !pto.tile_buf) + pto.tstore ins(%t0 : !pto.tile_buf) outs(%dp : !pto.partition_tensor_view<16x16xf16>) + + pto.tload ins(%sp : !pto.partition_tensor_view<16x16xf16>) outs(%t1 : !pto.tile_buf) + pto.tadd ins(%t1, %t1 : !pto.tile_buf, !pto.tile_buf) outs(%t1 : !pto.tile_buf) + pto.tstore ins(%t1 : !pto.tile_buf) outs(%dp : !pto.partition_tensor_view<16x16xf16>) + + pto.tload ins(%sp : !pto.partition_tensor_view<16x16xf16>) outs(%t2 : !pto.tile_buf) + pto.tadd ins(%t2, %t2 : !pto.tile_buf, !pto.tile_buf) outs(%t2 : !pto.tile_buf) + pto.tstore ins(%t2 : !pto.tile_buf) outs(%dp : !pto.partition_tensor_view<16x16xf16>) + + pto.tload ins(%sp : !pto.partition_tensor_view<16x16xf16>) outs(%t3 : !pto.tile_buf) + pto.tadd ins(%t3, %t3 : !pto.tile_buf, !pto.tile_buf) outs(%t3 : !pto.tile_buf) + pto.tstore ins(%t3 : !pto.tile_buf) outs(%dp : !pto.partition_tensor_view<16x16xf16>) + } + return + } +} diff --git a/test/lit/pto/auto_bufid_dual_subcore.pto b/test/lit/pto/auto_bufid_dual_subcore.pto new file mode 100644 index 0000000000..3079487923 --- /dev/null +++ b/test/lit/pto/auto_bufid_dual_subcore.pto @@ -0,0 +1,14 @@ +// A5 splits the vector core into two subcores, so a kernel_kind=vector kernel +// dispatches to both. Those are declined: measured 3 win / 3 lose, and nothing +// static separates them, so the gate stays conservative. +// +// RUN: ptoas --pto-arch=a5 --enable-insert-sync --auto-bufid-single-core %s 2>&1 \ +// RUN: | FileCheck %s + +module attributes {"pto.device-spec" = "Ascend910B1"} { + func.func @vector_dual_subcore() -> () attributes {pto.kernel_kind = #pto.kernel_kind} { + return + } +} + +// CHECK: [auto-bufid] dispatch=dual-subcore arch=a5 decision=events (declined) diff --git a/test/lit/pto/auto_bufid_single_core.pto b/test/lit/pto/auto_bufid_single_core.pto new file mode 100644 index 0000000000..6a2647e3fc --- /dev/null +++ b/test/lit/pto/auto_bufid_single_core.pto @@ -0,0 +1,19 @@ +// Kernel-entire buffer-id routing: a single-core (cube) dispatch is converted, +// a dual-subcore (vector) dispatch is declined and left on events. +// +// RUN: ptoas --pto-arch=a5 --enable-insert-sync --auto-bufid-single-core %s 2>&1 \ +// RUN: | FileCheck --check-prefix=CUBE %s +// RUN: ptoas --pto-arch=a3 --enable-insert-sync --auto-bufid-single-core %s 2>&1 \ +// RUN: | FileCheck --check-prefix=A3 %s + +module attributes {"pto.device-spec" = "Ascend910B1"} { + func.func @cube_single_core() -> () attributes {pto.kernel_kind = #pto.kernel_kind} { + return + } +} + +// A cube kernel dispatches to one core, so it is eligible. +// CUBE: [auto-bufid] dispatch=single-core arch=a5 decision=BUFFER-ID + +// Buffer-id does not exist on a3: always decline, whatever the dispatch. +// A3: [auto-bufid] dispatch=single-core arch=a3 decision=events (declined) diff --git a/test/lit/pto/sync_addr_reuse_diag.pto b/test/lit/pto/sync_addr_reuse_diag.pto new file mode 100644 index 0000000000..21c443c33a --- /dev/null +++ b/test/lit/pto/sync_addr_reuse_diag.pto @@ -0,0 +1,53 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===----------------------------------------------------------------------===// +// ADDRESS-REUSE DIAGNOSTIC. +// +// Reports orderings whose two endpoints are distinct tile allocations placed at one +// address, and classifies them by whether the two tiles have the same type: +// +// * reuse-ordering -- identical types. One buffer holding successive values, so +// the ordering is an artifact of the reuse rather than of dataflow. +// * alias-view-ordering -- differing types over the same bytes, a reshape or +// transpose view. The ordering carries real dataflow and must be kept. +// +// The distinction is the whole point of the check: counting the two together would +// attribute recoverable cost to orderings that are not removable at all. +// +// The type test is a heuristic. Two same-typed tiles can be aliased on purpose, and +// then the ordering between them is a real read-after-write through memory. The +// diagnostic cannot tell that from reuse, which is why it only reports. +// +// This test fails if the classification inverts, if either report is dropped, or if +// the diagnostic starts firing with the flag absent. It does not pin cycle cost -- +// the diagnostic reports and never fails a compile. +//===----------------------------------------------------------------------===// + +// RUN: ptoas --pto-arch=a3 --pto-level=level3 --enable-unified-sync --check-addr-reuse-war %S/Inputs/addr_reuse_war.pto 2>&1 | FileCheck %s +// RUN: ptoas --pto-arch=a3 --pto-level=level3 --enable-unified-sync %S/Inputs/addr_reuse_war.pto 2>&1 | FileCheck %s --check-prefix=OFF --allow-empty + +// One of each class, and one distinct reused address. `alias_view_deps` is counted +// separately and deliberately excluded from `orderings_from_reuse`. +// CHECK: [addr-reuse-war] func=addr_reuse_war orderings_from_reuse=1 reuse_addresses=1 alias_view_deps=1 + +// Differing types (1x64 against 8x8) at addr 512, so this is the alias view. +// CHECK: !! alias-view-ordering h#{{[0-9]+}} scope=vec addr=512 V->MTE2 +// CHECK-NEXT: first alloc loc("v0" +// CHECK-NEXT: second alloc loc("v1" + +// Identical types at addr 0, so this is the reuse. Allocations print in program +// order, not in the order the endpoints happened to match. +// CHECK: !! reuse-ordering h#{{[0-9]+}} scope=vec addr=0 MTE3->MTE2 +// CHECK-NEXT: first alloc loc("t0" +// CHECK-NEXT: second alloc loc("t1" + +// Off by default: no report surface at all without the flag. +// OFF-NOT: addr-reuse-war +// OFF-NOT: reuse-ordering +// OFF-NOT: alias-view-ordering diff --git a/test/lit/pto/sync_coverage_profile_version.pto b/test/lit/pto/sync_coverage_profile_version.pto new file mode 100644 index 0000000000..ceede3d8e1 --- /dev/null +++ b/test/lit/pto/sync_coverage_profile_version.pto @@ -0,0 +1,46 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===----------------------------------------------------------------------===// +// PERSISTED COVERAGE PROFILES CARRY A DIGEST VERSION. +// +// The G1 differential gate writes a profile in one ptoas process and compares it in +// another, so the anchor signature has to be reproducible across processes. It once used +// llvm::hash_value, which seeds itself from a function address when +// LLVM_ENABLE_ABI_BREAKING_CHECKS is on -- so under an assertion build ASLR changed the +// digest every run and identical source compared unequal. It is xxh3 now. +// +// That change makes every previously written profile incomparable. This test pins that +// such a profile is refused EXPLICITLY rather than being silently mistaken for a changed +// kernel. +// +// The stored profile below is deliberately wrong in BOTH ways: it has no sigver field +// (an older build wrote it) and its signature does not match this kernel either. The +// version verdict must win, because that is the difference between telling someone to +// regenerate a stale file and telling them their program changed. +//===----------------------------------------------------------------------===// + +// RUN: not ptoas --pto-arch=a3 --enable-insert-sync --check-sync-coverage=%S/Inputs/coverage_profile_v1.cov %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=STALE + +// Refused by version, naming both versions so the fix is obvious from the message alone. +// STALE: coverage-profile-version-mismatch +// STALE: signature version 0 but this build writes version 2 +// STALE: regenerate it with --dump-sync-coverage + +// And NOT blamed on the kernel, even though the stored signature is also wrong. +// STALE-NOT: anchor-signature-mismatch + +// A profile this build wrote round-trips, so the version check is not simply refusing +// everything. +// RUN: ptoas --pto-arch=a3 --enable-insert-sync --dump-sync-coverage %S/Inputs/oracle_vadd.pto 2> %t.fresh > /dev/null +// RUN: FileCheck %s --check-prefix=FRESH --input-file=%t.fresh +// FRESH: [sync-cov] func=oracle_vadd anchors=4 sig=0x{{[0-9a-f]+}} {{.*}} sigver=2 + +// RUN: ptoas --pto-arch=a3 --enable-insert-sync --check-sync-coverage=%t.fresh %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=ROUNDTRIP --allow-empty +// ROUNDTRIP-NOT: coverage-profile-version-mismatch +// ROUNDTRIP-NOT: anchor-signature-mismatch diff --git a/test/lit/pto/sync_event_rotation.pto b/test/lit/pto/sync_event_rotation.pto new file mode 100644 index 0000000000..a24ab0d36e --- /dev/null +++ b/test/lit/pto/sync_event_rotation.pto @@ -0,0 +1,114 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===----------------------------------------------------------------------===// +// EVENT ROTATION (d > 1): the unified allocator no longer collapses multi-buffered +// hazards to a single event id. +// +// THE SHAPE. On a `pto.alloc_multi_tile count=N` + `pto.multi_tile_get %mb[%slot]` +// loop the frontend hands the hazard `eventIdNum = N` and a slot SSA, and codegen +// emits `set_flag_dyn`/`wait_flag_dyn` selecting `eventIds[slot % N]`. A colourer +// that assigns one id per hazard collapses this: codegen's `eventIds.size() == 1` +// test then routes it to the STATIC emitter and every dyn op disappears. This is an +// event-path property, independent of buffer-ID routing. +// +// THE OPTIMALITY CLAIM HAD TO BE RESTATED, not just the code changed. Rotation +// makes the colouring WEIGHTED -- a hazard needs d ids live across its interval -- +// so the unit-demand "chi = omega" no longer holds as written. What holds is +// `chi = omega_weighted`, the peak SUM of demands over a point. That survives ONLY +// because the d ids need not be CONTIGUOUS, which was settled from code: +// - `CreateSetWaitOpForMultiBuffer` materialises each `eventIds[i]` as an +// independent constant picked by `slotMod == i`; +// - `set_flag_dyn` takes the id as a runtime `Index` operand and +// `LowerPipeEventDynSyncOpPattern` passes it straight through. No verifier, no +// lowering, no ISA text constrains the id set. +// Had contiguity been required this would be interval colouring with BANDWIDTH -- +// NP-hard in general -- and the greedy core would not have extended. +// +// THE COMPENSATION MUST EMIT d FLAGS, NOT ONE -- the iteration-1 deadlock in a +// rotation costume. The head/tail are synthesised and carry no slot SSA, so they hit +// the static fallback and would prime only `eventIds[0]`. The in-body +// `wait_flag_dyn` waits on `eventIds[slot % N]`, so the first iteration landing on +// slot != 0 would BLOCK FOREVER. Both ids must be primed before the loop. +//===----------------------------------------------------------------------===// + +// RUN: ptoas --pto-arch=a5 --enable-unified-sync --enable-unified-sync-debug --mlir-print-ir-after=pto-unified-sync %s 2>&1 >/dev/null | FileCheck %s + +// EVERY allocated id is primed before the loop -- the anti-deadlock invariant. +// CHECK: pto.set_flag[, , ] +// CHECK: pto.set_flag[, , ] +// CHECK: scf.for +// The in-body pair is slot-keyed, so it rotates between those two ids. +// CHECK: arith.select +// CHECK: pto.wait_flag_dyn[, , +// CHECK: pto.set_flag_dyn[, , + +// --------------------------------------------------------------------------- +// G1-self MODELS THE ROTATING PAIR, AT DISTANCE N. +// +// It used to be blind: `computeCoverage`'s type chain had no Dyn case, so the +// rotating pair fell off the end and contributed NO edge, and G1-self reported its +// ordering uncovered on BOTH modes (identical counts -- the same shared code). +// +// The distance is N, DERIVED not assumed: `wait_flag_dyn` in iteration j waits on +// eventIds[j % N] and `set_flag_dyn` in iteration k sets eventIds[k % N], so the +// wait at j is satisfied by the latest k < j with k%N == j%N -- i.e. k = j - N. +// N is recovered from the selector's own `arith.remui` modulus. +// +// THE EXPECTED SIDE HAD TO MOVE TOO, and in the WIDEN direction only. With N +// slots, iteration k+1 touches a DIFFERENT slot, so the d=1 requirement the check +// used to make was SPURIOUS -- the real conflict is at slot reuse, k+N. The +// dep is real; only its distance moves. This gate does not drop it, and there is +// no skip on that path -- dropping it would let an unrotated kernel pass. +// RUN: ptoas --pto-arch=a5 --enable-unified-sync --enable-unified-sync-debug --dump-sync-coverage %s 2>&1 >/dev/null | FileCheck %s --check-prefix=COV +// COV: [sync-cov] func=rot_d2_t1 anchors=2 {{.*}} bedge_count=2 bedges=1-0@2,1-1@1 +// +// `1-0@2` is the rotating edge: tstore (anchor#1) in iteration k before tload +// (anchor#0) in iteration k+2. `1-1@1` is the in-body PIPE_MTE3 barrier, which +// orders every ADJACENT iteration and so stays at d=1. +// +// With it modelled, the absolute gate closes on this kernel. +// RUN: ptoas --pto-arch=a5 --enable-unified-sync --enable-unified-sync-debug --check-sync-self-coverage %s 2>&1 >/dev/null | FileCheck %s --check-prefix=SELF +// SELF: [sync-gate:g1-self] func=rot_d2_t1 compounds=2 deps=4 covered=4 uncovered=0 (samepipe=0 crosspipe=0) carried=3/3 arm_excluded=0 arch_guaranteed=0 pipe_self_ordered=0 unmappable=0 OK +// +// NON-VACUITY, and this is the check that catches "dropped" masquerading as +// "covered": with NO sync pass the SAME kernel must still DEMAND the carried +// dependency and report it uncovered. `deps=2` is unchanged from the green run +// above -- the dependency was re-distanced, never excused. +// RUN: not ptoas --pto-arch=a5 --check-sync-self-coverage %s 2>&1 >/dev/null | FileCheck %s --check-prefix=NOSYNC +// NOSYNC: [sync-gate:g1-self] func=rot_d2_t1 compounds=2 deps=4 covered=0 uncovered=4 (samepipe=2 crosspipe=2) carried=3/0 +// NOSYNC: !! kind=uncovered-dependency war/carried anchor#1 (pto.tstore) -> anchor#0 (pto.tload) PIPE_MTE3->PIPE_MTE2 + + +// --------------------------------------------------------------------------- +// THE ROUTING PREDICATE'S OVERFLOW PREDICTION IS WEIGHTED BY d. +// +// A hazard with demand d holds d ids live across its interval, so the overflow +// prediction must sum demands rather than count hazards. Unweighted, it +// under-predicts pressure by a factor of d and declines to route exactly the buffers +// the colourer then overflows and spills to PIPE_ALL barriers. This kernel must +// therefore reach the colourer with nothing overflowing and nothing spilled. +// RUN: ptoas --pto-arch=a5 --enable-unified-sync --enable-unified-sync-debug %s 2>&1 >/dev/null | FileCheck %s --check-prefix=WEIGHTED +// WEIGHTED: [unified-sync coloring] interval coloring: assigned={{[0-9]+}} overflow=0 spilled_to_barrier=0 + +module attributes {pto.target_arch = "a5"} { + func.func @rot_d2_t1(%gm : !pto.partition_tensor_view<16x16xf16>, + %dst : !pto.partition_tensor_view<16x16xf16>, %n : index) { + %c0 = arith.constant 0 : index + %c1 = arith.constant 1 : index + %cN = arith.constant 2 : index + %mb0 = pto.alloc_multi_tile : !pto.multi_tile_buf + scf.for %i = %c0 to %n step %c1 { + %idx = arith.remui %i, %cN : index + %p0 = pto.multi_tile_get %mb0[%idx] : !pto.multi_tile_buf -> !pto.tile_buf + pto.tload ins(%gm : !pto.partition_tensor_view<16x16xf16>) outs(%p0 : !pto.tile_buf) + pto.tstore ins(%p0 : !pto.tile_buf) outs(%dst : !pto.partition_tensor_view<16x16xf16>) + } + return + } +} diff --git a/test/lit/pto/sync_extract_oracle_views.pto b/test/lit/pto/sync_extract_oracle_views.pto new file mode 100644 index 0000000000..086a0a05b6 --- /dev/null +++ b/test/lit/pto/sync_extract_oracle_views.pto @@ -0,0 +1,66 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===----------------------------------------------------------------------===// +// FORMAT PIN FOR THE SHARED SYNC-OP EXTRACTOR. +// +// The oracle's gates read one of two views of the same synchronization, so both +// renderings must stay stable and field-correct: +// +// * IR view (post-codegen): the emitted pto.set_flag / pto.wait_flag / +// pto.get_buf / pto.rls_buf / pto.barrier ops. This is the surface for the +// differential gates -- coverage superset and sync-count floor -- and for +// device-id legality. +// * SyncOperation view (pre-codegen): the in-memory set the allocator assigns ids +// into. This is the surface for the in-pass gates. +// +// This test fails if a field is dropped or renamed, if program order changes, or if +// the two views stop agreeing on a kernel's hazard structure. The two are not +// redundant: the IR view cannot see a mechanism or a pre-codegen id, and the syncops +// view cannot see what was actually emitted. +//===----------------------------------------------------------------------===// + +// RUN: ptoas --pto-arch=a3 --enable-insert-sync --dump-sync-extract %S/Inputs/oracle_vadd.pto 2>&1 | FileCheck %s --check-prefix=IRVIEW +// RUN: ptoas --pto-arch=a3 --enable-unified-sync --enable-unified-sync-debug %S/Inputs/oracle_vadd.pto 2>&1 | FileCheck %s --check-prefix=SYNCOPS + +// IR view: two set/wait pairs (MTE2->V, V->MTE3) plus the tail PIPE_ALL barrier. +// Pipes, event_id and program order are all pinned. Note the numbering: the IR view +// counts OPS, so a pair occupies two indices. +// IRVIEW: sync-extract:ir{{.*}} func=oracle_vadd records=5 +// IRVIEW-NEXT: #0 pto.set_flag src=PIPE_MTE2 dst=PIPE_V event_id=0 +// IRVIEW-NEXT: #1 pto.wait_flag src=PIPE_MTE2 dst=PIPE_V event_id=0 +// IRVIEW-NEXT: #2 pto.set_flag src=PIPE_V dst=PIPE_MTE3 event_id=0 +// IRVIEW-NEXT: #3 pto.wait_flag src=PIPE_V dst=PIPE_MTE3 event_id=0 +// IRVIEW-NEXT: #4 pto.barrier src=PIPE_ALL dst=PIPE_ALL + +// SyncOperation view: the same three hazards, but numbered by HAZARD rather than by +// op, so both halves of a pair share an index. That difference is the reason both +// views are pinned here together -- a gate reading one cannot infer the other's +// indices. +// +// `mech` is the resource class backing each sync, DERIVED FROM TYPE at construction. +// That is why the tail reads `barrier` and not `event`: a flat default of event would +// make every barrier claim an event id it does not hold. Nothing here reads `bufid`, +// because this kernel's buffers do not overflow their event pool and the router +// declines them -- sync_overflow_id_gap.pto covers a kernel where it does not, and +// sync_mechanism_field.pto forces the value directly. +// +// `event_ids` are populated by the allocator, and two properties are visible: +// - BOTH halves of a pair carry the SAME id. The id, like the mechanism, belongs to +// the hazard and is never stamped on a lone SyncOperation. +// - both pairs use id 0, because event flags are per-(src,dst) DISJOINT hardware: +// id 0 in MTE2->V is a different flag from id 0 in V->MTE3. This matches what +// InsertSync emits on this kernel. +// The barrier carries no id, having none to carry. +// SYNCOPS: unified-sync model{{.*}} kernel=oracle_vadd K=0 hazards=3 +// SYNCOPS: sync-extract:syncops{{.*}} func=oracle_vadd records=5 +// SYNCOPS-NEXT: #0 set_flag mech=event src=MTE2 dst=V irIdx=1 event_ids=[0] +// SYNCOPS-NEXT: #0 wait_flag mech=event src=MTE2 dst=V irIdx=2 event_ids=[0] +// SYNCOPS-NEXT: #1 set_flag mech=event src=V dst=MTE3 irIdx=2 event_ids=[0] +// SYNCOPS-NEXT: #1 wait_flag mech=event src=V dst=MTE3 irIdx=3 event_ids=[0] +// SYNCOPS-NEXT: #2 pipe_barrier mech=barrier src=ALL dst=ALL irIdx=3 event_ids=[] diff --git a/test/lit/pto/sync_forced_overflow.pto b/test/lit/pto/sync_forced_overflow.pto new file mode 100644 index 0000000000..5637fb2398 --- /dev/null +++ b/test/lit/pto/sync_forced_overflow.pto @@ -0,0 +1,336 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===----------------------------------------------------------------------===// +// FORCED EVENT-ID OVERFLOW. +// +// WHY THIS FILE EXISTS AT ALL. No real kernel overflows: per-direction peak overlap +// stays inside the 8-id pool, so the routing predicate ("does this buffer overflow +// its pool?") would never fire on a real kernel, and neither would the spill path. +// Without a synthetic input that overflows on purpose, both would be machinery no +// test can execute. +// +// HOW IT FORCES OVERFLOW -- with a REAL KERNEL, not a test-only pool knob. All N +// tloads are issued before any consumer, so all N MTE2->V hazards are +// simultaneously live and the per-direction peak overlap is exactly N. The event +// pool for one direction is 8, so overflow = max(0, N - 8). +// +// IT IS A DIAL, not a single data point: overflow == max(0, N - 8) in N, so the +// harness can be moved to any point on the curve. Two points are pinned below -- +// N=8, the boundary, which must NOT overflow, and N=12, which must overflow by 4. +// +// BOTH DIRECTIONS ARE PINNED. A synthetic that always overflows proves nothing -- +// it could be broken rather than saturated. So the N=8 boundary kernel is pinned +// as a CONTROL that must NOT overflow. If OVERFLOW ever reports overflow=0, or +// BOUNDARY ever reports overflow>0, the dial has stopped being a dial. +// +// A5 (K=32) on purpose: buffer-ID is the class overflow is routed INTO, so +// the harness must run where K > 0. +//===----------------------------------------------------------------------===// + +// N=12: 12 live MTE2->V hazards against a pool of 8 -> overflow=4. +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-unified-sync --enable-unified-sync-debug %s 2>&1 >/dev/null | FileCheck %s --check-prefix=OVERFLOW +// OVERFLOW: [unified-model] func=forced_overflow_n12 hazards={{[0-9]+}} K=32 +// An overflowing hazard does not go out with no id: each spills to a PIPE_ALL +// barrier, so `overflow` and `spilled_to_barrier` must agree. +// +// THIS KERNEL IS THE ONLY REMAINING COVERAGE OF THE SPILL PATH. Written-back +// overflow kernels are ROUTED to buffer-id instead of spilled (see +// sync_overflow_id_gap.pto). This kernel is a pure +// forward MTE2->V chain -- never written back, so never routed -- which is what +// keeps the spill path exercised. If it ever starts routing, spilling becomes +// untested. If the two counts ever diverge, a hazard has been left unallocated. +// OVERFLOW: [unified-sync coloring] interval coloring: assigned=9 overflow=4 spilled_to_barrier=4 hidden_reserved=0 reused=0 +// +// N=8 CONTROL: exactly the pool. MUST NOT overflow -- this is what proves the +// dial is a dial and not a kernel that is simply always broken. +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-unified-sync --enable-unified-sync-debug %s 2>&1 >/dev/null | FileCheck %s --check-prefix=BOUNDARY +// BOUNDARY: [unified-model] func=boundary_n8 hazards={{[0-9]+}} K=32 +// BOUNDARY: [unified-sync coloring] interval coloring: assigned=9 overflow=0 spilled_to_barrier=0 hidden_reserved=0 reused=0 + +// --------------------------------------------------------------------------- +// STEP 2 ROUTING. +// +// EVERY CHECK BELOW IS ANCHORED ON `func=`, AND THAT IS LOAD-BEARING. Three +// functions share this module and the per-function report lines are emitted in a +// non-deterministic FUNCTION order. Each function's own lines are contiguous, so a +// check anchored on `func=` and the lines that follow it are sound -- but an +// unanchored check is not: two of these kernels report the same `considered=13`, so +// without the function name the match is ambiguous and the test flakes on order. +// +// The two kernels above force OVERFLOW but NOT WRITE-BACK, and the routing predicate +// needs BOTH: a pure forward MTE2->V chain leaves isWrittenBack false, giving +// `routed=0 skipped(not_written_back=12)`. `overflow_writeback` below supplies the +// second dimension, so overflow alone is never mistaken for a routing condition. +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-unified-sync --enable-unified-sync-debug %s 2>&1 >/dev/null | FileCheck %s --check-prefix=NOROUTE +// NOROUTE: [unified-sync routing] func=forced_overflow_n12 buffer routing: considered=13 routed=0 hazards_covered=0 skipped(no_overflow=1 not_written_back=12 K=0:0) not_expressible=0 split=0 OK +// +// With write-back, 12 of 13 buffers route. The 13th is the output tile: its +// direction does not overflow, so `no_overflow=1` -- the predicate discriminating, +// not routing everything in sight. `split=0` is the ALL-OR-NOTHING assertion: no +// hazard touches both a routed and an unrouted buffer, which is the shape that hangs. +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-unified-sync --enable-unified-sync-debug %s 2>&1 >/dev/null | FileCheck %s --check-prefix=ROUTE +// ROUTE: [unified-sync routing] func=overflow_writeback buffer routing: considered=13 routed=12 hazards_covered=24 skipped(no_overflow=1 not_written_back=0 K=0:0) not_expressible=0 split=0 OK +// ROUTE-NEXT: ROUTE overflow_writeback b#0 -> bufid omega=12>pool=8 wb=yes +// +// AND THE EMISSION HAPPENS: the routed hazards take no event +// id and get get_buf/rls_buf instead. `get` == `rls` is the counter cycle closing. +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-unified-sync --enable-unified-sync-debug %s 2>&1 >/dev/null | FileCheck %s --check-prefix=EMITWB +// EMITWB: [unified-sync bufid] func=overflow_writeback bufid emission: routed_buffers=12 routed_hazards=24 anchor_ops=24 get=24 rls=24 physical_ids=13 nesting=OK emit=OK bracketed_hazards=24 unbracketed_hazards=0 suppressed_event_ops=72 leaked_event_ops=0 ALL-OR-NOTHING-OK +// +// ALL-OR-NOTHING is asserted by the compiler itself, per function, in the bufid +// emission line above: `leaked_event_ops=0`. A whole-file `CHECK-NOT` was tried first and is +// WRONG here -- the IR dump holds all three functions and the two UNROUTED ones +// legitimately emit MTE2<->V flags, so it flagged them. Per-function assertion in +// the report is both correct and immune to the report-ordering flake. +// +// sync_overflow_id_gap.pto pins the same spill from the other direction, and stays +// single-function, because a `not ptoas` check cannot +// live in a multi-function module: when the pass fails on the first function it +// processes, `signalPassFailure` stops the pipeline and the other functions' reports +// are ABSENT, not merely reordered. +// +// G1 AND THE BARRIER DELTA. The differential coverage gate FAILS on this kernel, and +// that failure is the intended reading rather than an open defect. Routing twelve +// buffers to tokens keeps the event pool from overflowing, so this run emits NO +// in-body PIPE_ALL where the incumbent emits eight. A PIPE_ALL orders every anchor +// pair spanning it, so the incumbent's edge set is denser by construction and the +// superset requirement cannot be satisfied by emitting less sync. The two functions +// above, whose barrier counts match, are clean in the very same run -- the barrier +// delta is the entire discriminator. +// +// The gate reports the two barrier counts and concludes nothing from them, because +// the same delta appears whenever the candidate is simply broken -- an empty +// allocation shows it too, the reference carrying the auto-sync tail barrier. So the +// counts are pinned here as facts and the reading is argued in the header. +// +// Three checks hold this down, and the middle one is what makes it evidence instead +// of an excuse: +// G1FWD the failure, carrying the barrier counts it is attributed to +// G1REV the same comparison with the two profiles SWAPPED passes, so this +// run's edge set is a strict SUBSET -- there is no ordering it +// establishes that the incumbent lacks. That is what separates +// "emitted less" from "diverged", and it is the direction a real +// regression could not fake +// SELFU / SELFI the absolute gate is clean for BOTH modes, which is the +// correctness claim the differential is unable to make +// +// Only `func=overflow_writeback` lines are asserted, for the reason recorded just +// above: a failing function can cost the remaining functions their reports, and +// report order across functions is not stable. +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-insert-sync --dump-sync-coverage %s 2> %t.ref.cov > /dev/null +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-unified-sync --dump-sync-coverage %s 2> %t.cand.cov > /dev/null +// RUN: not ptoas --pto-arch=a5 --pto-level=level3 --enable-unified-sync --check-sync-coverage=%t.ref.cov %s 2>&1 >/dev/null | FileCheck %s --check-prefix=G1FWD +// G1FWD: [sync-gate:g1] func=overflow_writeback candidate_edges=90 reference_edges=236 violations=665 candidate_bedges=90 reference_bedges=609 candidate_barrier_all=1 reference_barrier_all=9 +// +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-insert-sync --check-sync-coverage=%t.cand.cov %s 2>&1 >/dev/null | FileCheck %s --check-prefix=G1REV +// G1REV: [sync-gate:g1] func=overflow_writeback candidate_edges=236 reference_edges=90 violations=0 OK +// +// THE ABSOLUTE GATE, BOTH MODES. `arch_guaranteed` counts dependencies left to the A5 +// PIPE_V intra-pipe guarantee instead of to emitted sync, and it is pinned on BOTH +// sides because this run leans on it harder -- 66 against 28. Those sixty-six are the +// WAW pairs among the twelve `tcvt` writes into ONE tile buffer, which program order +// already orders on every profile; the two modes discharge different numbers of +// those pairs (114 covered here against 152), and `covered` is tested before the +// carve-out. Both sides +// reach uncovered=0, so what differs is which mechanism discharges a pair, not whether +// it is discharged. Pinning both numbers makes a future shift in that balance visible. +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-unified-sync --check-sync-self-coverage %s 2>&1 >/dev/null | FileCheck %s --check-prefix=SELFU +// SELFU: [sync-gate:g1-self] func=overflow_writeback compounds=25 deps=205 covered=139 uncovered=0 (samepipe=0 crosspipe=0) carried=115/115 arm_excluded=0 arch_guaranteed=66 pipe_self_ordered=0 unmappable=0 OK +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-insert-sync --check-sync-self-coverage %s 2>&1 >/dev/null | FileCheck %s --check-prefix=SELFI +// SELFI: [sync-gate:g1-self] func=overflow_writeback compounds=25 deps=205 covered=177 uncovered=0 (samepipe=0 crosspipe=0) carried=115/115 arm_excluded=0 arch_guaranteed=28 pipe_self_ordered=0 unmappable=0 OK +// +// AND THE COUNT FLOOR PASSES THE SAME FACT the coverage gate reports as loss: 55 ops +// against 63. Two gates reading one reduction in opposite directions is the clearest +// statement of what each one measures. +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-insert-sync --dump-sync-counts %s 2> %t.ref.cnt > /dev/null +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-unified-sync --check-sync-count-floor=%t.ref.cnt %s 2>&1 >/dev/null | FileCheck %s --check-prefix=G4 +// G4: [sync-gate:g4] func=overflow_writeback candidate_total=55 reference_total=63 violations=0 OK + + +module attributes {pto.target_arch = "a5"} { + func.func @forced_overflow_n12(%src: !pto.ptr, %dst: !pto.ptr) attributes {pto.kernel_kind = #pto.kernel_kind} { + %c0 = arith.constant 0 : index + %c1 = arith.constant 1 : index + %c32 = arith.constant 32 : index + %a0 = arith.constant 0 : i64 + %a1 = arith.constant 2048 : i64 + %a2 = arith.constant 4096 : i64 + %a3 = arith.constant 6144 : i64 + %a4 = arith.constant 8192 : i64 + %a5 = arith.constant 10240 : i64 + %a6 = arith.constant 12288 : i64 + %a7 = arith.constant 14336 : i64 + %a8 = arith.constant 16384 : i64 + %a9 = arith.constant 18432 : i64 + %a10 = arith.constant 20480 : i64 + %a11 = arith.constant 22528 : i64 + %aout = arith.constant 24576 : i64 + %sv = pto.make_tensor_view %src, shape = [%c32, %c32], strides = [%c32, %c1] : !pto.tensor_view + %dv = pto.make_tensor_view %dst, shape = [%c32, %c32], strides = [%c32, %c1] : !pto.tensor_view + %ps = pto.partition_view %sv, offsets = [%c0, %c0], sizes = [%c32, %c32] : !pto.tensor_view -> !pto.partition_tensor_view<32x32xf16> + %pd = pto.partition_view %dv, offsets = [%c0, %c0], sizes = [%c32, %c32] : !pto.tensor_view -> !pto.partition_tensor_view<32x32xf16> + %t0 = pto.alloc_tile addr = %a0 : !pto.tile_buf + %t1 = pto.alloc_tile addr = %a1 : !pto.tile_buf + %t2 = pto.alloc_tile addr = %a2 : !pto.tile_buf + %t3 = pto.alloc_tile addr = %a3 : !pto.tile_buf + %t4 = pto.alloc_tile addr = %a4 : !pto.tile_buf + %t5 = pto.alloc_tile addr = %a5 : !pto.tile_buf + %t6 = pto.alloc_tile addr = %a6 : !pto.tile_buf + %t7 = pto.alloc_tile addr = %a7 : !pto.tile_buf + %t8 = pto.alloc_tile addr = %a8 : !pto.tile_buf + %t9 = pto.alloc_tile addr = %a9 : !pto.tile_buf + %t10 = pto.alloc_tile addr = %a10 : !pto.tile_buf + %t11 = pto.alloc_tile addr = %a11 : !pto.tile_buf + %tout = pto.alloc_tile addr = %aout : !pto.tile_buf + // All 12 loads first: opens 12 simultaneous MTE2->V hazards. + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t0 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t1 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t2 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t3 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t4 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t5 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t6 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t7 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t8 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t9 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t10 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t11 : !pto.tile_buf) + // Then all consumers: every one of those intervals is still live. + pto.tcvt ins(%t0{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t1{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t2{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t3{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t4{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t5{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t6{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t7{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t8{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t9{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t10{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t11{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tstore ins(%tout : !pto.tile_buf) outs(%pd : !pto.partition_tensor_view<32x32xf16>) + return + } + func.func @boundary_n8(%src: !pto.ptr, %dst: !pto.ptr) attributes {pto.kernel_kind = #pto.kernel_kind} { + %c0 = arith.constant 0 : index + %c1 = arith.constant 1 : index + %c32 = arith.constant 32 : index + %a0 = arith.constant 0 : i64 + %a1 = arith.constant 2048 : i64 + %a2 = arith.constant 4096 : i64 + %a3 = arith.constant 6144 : i64 + %a4 = arith.constant 8192 : i64 + %a5 = arith.constant 10240 : i64 + %a6 = arith.constant 12288 : i64 + %a7 = arith.constant 14336 : i64 + %aout = arith.constant 16384 : i64 + %sv = pto.make_tensor_view %src, shape = [%c32, %c32], strides = [%c32, %c1] : !pto.tensor_view + %dv = pto.make_tensor_view %dst, shape = [%c32, %c32], strides = [%c32, %c1] : !pto.tensor_view + %ps = pto.partition_view %sv, offsets = [%c0, %c0], sizes = [%c32, %c32] : !pto.tensor_view -> !pto.partition_tensor_view<32x32xf16> + %pd = pto.partition_view %dv, offsets = [%c0, %c0], sizes = [%c32, %c32] : !pto.tensor_view -> !pto.partition_tensor_view<32x32xf16> + %t0 = pto.alloc_tile addr = %a0 : !pto.tile_buf + %t1 = pto.alloc_tile addr = %a1 : !pto.tile_buf + %t2 = pto.alloc_tile addr = %a2 : !pto.tile_buf + %t3 = pto.alloc_tile addr = %a3 : !pto.tile_buf + %t4 = pto.alloc_tile addr = %a4 : !pto.tile_buf + %t5 = pto.alloc_tile addr = %a5 : !pto.tile_buf + %t6 = pto.alloc_tile addr = %a6 : !pto.tile_buf + %t7 = pto.alloc_tile addr = %a7 : !pto.tile_buf + %tout = pto.alloc_tile addr = %aout : !pto.tile_buf + // All 8 loads first: opens 8 simultaneous MTE2->V hazards. + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t0 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t1 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t2 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t3 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t4 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t5 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t6 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t7 : !pto.tile_buf) + // Then all consumers: every one of those intervals is still live. + pto.tcvt ins(%t0{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t1{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t2{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t3{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t4{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t5{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t6{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t7{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tstore ins(%tout : !pto.tile_buf) outs(%pd : !pto.partition_tensor_view<32x32xf16>) + return + } + // The loop is what creates the WRITE-BACK: each tile is written by MTE2 and + // read by V on every iteration, so it carries BOTH MTE2->V (forward RAW) and + // V->MTE2 (loop-carried WAR). That makes isWrittenBack true, which the routing + // predicate requires and which the two kernels above do NOT have. + func.func @overflow_writeback(%src: !pto.ptr, %dst: !pto.ptr) attributes {pto.kernel_kind = #pto.kernel_kind} { + %c0 = arith.constant 0 : index + %c1 = arith.constant 1 : index + %c32 = arith.constant 32 : index + %c4 = arith.constant 4 : index + %a0 = arith.constant 0 : i64 + %a1 = arith.constant 2048 : i64 + %a2 = arith.constant 4096 : i64 + %a3 = arith.constant 6144 : i64 + %a4 = arith.constant 8192 : i64 + %a5 = arith.constant 10240 : i64 + %a6 = arith.constant 12288 : i64 + %a7 = arith.constant 14336 : i64 + %a8 = arith.constant 16384 : i64 + %a9 = arith.constant 18432 : i64 + %a10 = arith.constant 20480 : i64 + %a11 = arith.constant 22528 : i64 + %aout = arith.constant 24576 : i64 + %sv = pto.make_tensor_view %src, shape = [%c32, %c32], strides = [%c32, %c1] : !pto.tensor_view + %dv = pto.make_tensor_view %dst, shape = [%c32, %c32], strides = [%c32, %c1] : !pto.tensor_view + %ps = pto.partition_view %sv, offsets = [%c0, %c0], sizes = [%c32, %c32] : !pto.tensor_view -> !pto.partition_tensor_view<32x32xf16> + %pd = pto.partition_view %dv, offsets = [%c0, %c0], sizes = [%c32, %c32] : !pto.tensor_view -> !pto.partition_tensor_view<32x32xf16> + scf.for %it = %c0 to %c4 step %c1 { + %t0 = pto.alloc_tile addr = %a0 : !pto.tile_buf + %t1 = pto.alloc_tile addr = %a1 : !pto.tile_buf + %t2 = pto.alloc_tile addr = %a2 : !pto.tile_buf + %t3 = pto.alloc_tile addr = %a3 : !pto.tile_buf + %t4 = pto.alloc_tile addr = %a4 : !pto.tile_buf + %t5 = pto.alloc_tile addr = %a5 : !pto.tile_buf + %t6 = pto.alloc_tile addr = %a6 : !pto.tile_buf + %t7 = pto.alloc_tile addr = %a7 : !pto.tile_buf + %t8 = pto.alloc_tile addr = %a8 : !pto.tile_buf + %t9 = pto.alloc_tile addr = %a9 : !pto.tile_buf + %t10 = pto.alloc_tile addr = %a10 : !pto.tile_buf + %t11 = pto.alloc_tile addr = %a11 : !pto.tile_buf + %tout = pto.alloc_tile addr = %aout : !pto.tile_buf + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t0 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t1 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t2 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t3 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t4 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t5 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t6 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t7 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t8 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t9 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t10 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t11 : !pto.tile_buf) + pto.tcvt ins(%t0{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t1{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t2{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t3{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t4{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t5{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t6{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t7{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t8{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t9{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t10{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t11{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tstore ins(%tout : !pto.tile_buf) outs(%pd : !pto.partition_tensor_view<32x32xf16>) + scf.yield + } + return + } +} diff --git a/test/lit/pto/sync_gate_g1_buf_token_pipe.pto b/test/lit/pto/sync_gate_g1_buf_token_pipe.pto new file mode 100644 index 0000000000..c3a7db811a --- /dev/null +++ b/test/lit/pto/sync_gate_g1_buf_token_pipe.pto @@ -0,0 +1,37 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// A `get_buf`/`rls_buf` `op_type` has three legal spellings and G1-self must decode +// every one: an undecoded spelling drops the op from the coverage walk, and a dropped +// sync op is indistinguishable from an absent one, so the ordering the token +// establishes is reported as an uncovered dependency. +// +// The two inputs are the same chain over the same two pipes, differing only in +// spelling, so the pair is what makes the property testable -- either alone would +// pass a gate that decoded only that one spelling. +// +// `--emit-pto-ir` stops before the backend on purpose. `pto.get_buf` in input IR does +// not lower, and the property under test belongs to the gate, not to codegen. + +// The plain PipeAttr spelling. +// RUN: ptoas --pto-arch=a5 --check-sync-self-coverage --emit-pto-ir %S/Inputs/oracle_buf_token_spellings.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=PIPEATTR +// PIPEATTR: [sync-gate:g1-self] func=buf_chain_pipe_attr compounds=2 deps=1 covered=1 uncovered=0 (samepipe=0 crosspipe=0) carried=0/0 arm_excluded=0 arch_guaranteed=0 pipe_self_ordered=0 unmappable=0 OK + +// The pipe_event_type spelling, which decoded correctly already. Pinned so a +// regression that broke this one instead would not go unnoticed. +// RUN: ptoas --pto-arch=a5 --check-sync-self-coverage --emit-pto-ir %S/Inputs/oracle_buf_token_spellings.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=EVENTTYPE +// EVENTTYPE: [sync-gate:g1-self] func=buf_chain_event_type compounds=2 deps=1 covered=1 uncovered=0 (samepipe=0 crosspipe=0) carried=0/0 arm_excluded=0 arch_guaranteed=0 pipe_self_ordered=0 unmappable=0 OK + +// G2 and G3 read the same ops through the extractor, which already decoded all three +// spellings. Pinning them here records that the three gates agree on this input: a +// decode gap in one gate alone shows up as two gates accepting what a third rejects. +// RUN: ptoas --pto-arch=a5 --check-sync-interference --emit-pto-ir %S/Inputs/oracle_buf_token_spellings.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=G2 +// G2: [sync-gate:g2] func=buf_chain_pipe_attr view=ir records=4 errors=0 warnings=0 OK + +// RUN: ptoas --pto-arch=a5 --check-sync-ids --emit-pto-ir %S/Inputs/oracle_buf_token_spellings.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=G3 +// G3: [sync-gate:g3] func=buf_chain_pipe_attr view=ir records=4 violations=0 OK diff --git a/test/lit/pto/sync_gate_g1_coverage.pto b/test/lit/pto/sync_gate_g1_coverage.pto new file mode 100644 index 0000000000..e54c9505d4 --- /dev/null +++ b/test/lit/pto/sync_gate_g1_coverage.pto @@ -0,0 +1,41 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// G1 asserts that every ordering a reference compilation establishes is also +// established by the one under test. Unlike the count floor, it cannot be cleared +// by a compilation that emits nothing. + +// --------------------------------------------------------------------------- +// 1. The reference profile, and a run compared against itself. +// RUN: ptoas --pto-arch=a3 --enable-insert-sync --dump-sync-coverage %S/Inputs/oracle_vadd.pto 2> %t.ref > /dev/null +// RUN: FileCheck %s --check-prefix=PROFILE --input-file=%t.ref +// PROFILE: [sync-cov] func=oracle_vadd anchors=4 sig=0x{{[0-9a-f]+}} edge_count=3 edges=0-2,1-2,2-3 bedge_count=0 bedges= barrier_all=1 + +// RUN: ptoas --pto-arch=a3 --enable-insert-sync --check-sync-coverage=%t.ref %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=SELF +// SELF: [sync-gate:g1] func=oracle_vadd candidate_edges=3 reference_edges=3 violations=0 OK + +// --------------------------------------------------------------------------- +// 2. THE FAULT CLASS THIS GATE EXISTS FOR. Compiling with no synchronization pass +// at all emits nothing, which the count floor clears trivially. Every ordering +// is reported missing, and each names the anchor pair it lost. +// RUN: not ptoas --pto-arch=a3 --check-sync-coverage=%t.ref %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=EMPTY +// EMPTY: [sync-gate:g1] func=oracle_vadd candidate_edges=0 reference_edges=3 violations=3 +// EMPTY-NEXT: !! kind=missing-ordering anchor#0 -> anchor#2 +// EMPTY-NEXT: !! kind=missing-ordering anchor#1 -> anchor#2 +// EMPTY-NEXT: !! kind=missing-ordering anchor#2 -> anchor#3 + +// --------------------------------------------------------------------------- +// 3. THE BUFFER-TOKEN RULE IS LOAD-BEARING, and this is what shows it. On A5 the +// buffer-id pass discharges these hazards with get_buf/rls_buf instead of event +// flags. Its edge set is a strict SUPERSET of the event reference's, so the gate +// passes -- but only because `rls_buf(b)` is credited as ordering the following +// `get_buf(b)`. Drop that rule and this candidate appears to order nothing and +// the gate rejects a correct compilation. +// RUN: ptoas --pto-arch=a5 --enable-insert-sync --dump-sync-coverage %S/Inputs/oracle_vadd.pto 2> %t.a5 > /dev/null +// RUN: ptoas --pto-arch=a5 --enable-bufid_sync --check-sync-coverage=%t.a5 %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=BUFID +// BUFID: [sync-gate:g1] func=oracle_vadd candidate_edges=4 reference_edges=3 violations=0 OK diff --git a/test/lit/pto/sync_gate_g1_pipe_s_self_ordered.pto b/test/lit/pto/sync_gate_g1_pipe_s_self_ordered.pto new file mode 100644 index 0000000000..a104db1c87 --- /dev/null +++ b/test/lit/pto/sync_gate_g1_pipe_s_self_ordered.pto @@ -0,0 +1,35 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// G1-self excuses a PIPE_S -> PIPE_S dependency, on every arch. Neither allocator in +// this tree orders such a pair -- `IsNoNeedToInsertSync` returns before running the +// dependence analysis, and GraphSyncSolver's same-pipe path returns for PIPE_S +// unconditionally -- so requiring one here makes the gate unsatisfiable, not strict. +// The exclusion is COUNTED in its own field rather than folded into +// `arch_guaranteed`, because no hardware guarantee is claimed for it. +// +// The input is an existing kernel rather than a purpose-built one: this file's whole +// reason to exist is that `scalar_gm_tstore_sync.pto` compiles and runs while G1 +// rejected it, so the property is worth pinning on the real thing. + +// The dependency is excused and the kernel is clean. `deps=3` is `covered=2` plus the +// one excused pair, so the count still balances and nothing was dropped. +// RUN: ptoas --pto-arch=a3 --enable-insert-sync --check-sync-self-coverage %S/scalar_gm_tstore_sync.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=A3 +// A3: [sync-gate:g1-self] func=tstore_then_load_scalar_unknown_inttoptr compounds=4 deps=3 covered=2 uncovered=0 (samepipe=0 crosspipe=0) carried=0/0 arm_excluded=0 arch_guaranteed=0 pipe_self_ordered=1 unmappable=0 OK + +// THE SAME ON A5, which is the property that separates this exclusion from the +// PIPE_V one. That is gated on A5 because only A5's codegen declines the barrier; +// this is not gated, because both allocators decline PIPE_S on every arch. +// RUN: ptoas --pto-arch=a5 --enable-insert-sync --check-sync-self-coverage %S/scalar_gm_tstore_sync.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=A5 +// A5: [sync-gate:g1-self] func=tstore_then_load_scalar_unknown_inttoptr compounds=4 deps=3 covered=2 uncovered=0 (samepipe=0 crosspipe=0) carried=0/0 arm_excluded=0 arch_guaranteed=0 pipe_self_ordered=1 unmappable=0 OK + +// NOT WIDENED TO "ANYTHING TOUCHING PIPE_S". A scalar op ordered against a tstore is +// a cross-pipe pair, and it is still required and still covered by real sync -- +// `pipe_self_ordered=0` here is what shows the exclusion tests both endpoints. +// RUN: ptoas --pto-arch=a3 --enable-insert-sync --check-sync-self-coverage %S/scalar_gm_tstore_sync.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=CROSS +// CROSS: [sync-gate:g1-self] func=store_scalar_then_tstore compounds=3 deps=2 covered=2 uncovered=0 (samepipe=0 crosspipe=0) carried=0/0 arm_excluded=0 arch_guaranteed=0 pipe_self_ordered=0 unmappable=0 OK diff --git a/test/lit/pto/sync_gate_g1_self_coverage.pto b/test/lit/pto/sync_gate_g1_self_coverage.pto new file mode 100644 index 0000000000..3a7f3a7266 --- /dev/null +++ b/test/lit/pto/sync_gate_g1_self_coverage.pto @@ -0,0 +1,54 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// G1-self asks whether every dependency the front-end analysis reports is ordered +// by the emitted synchronization. It is ABSOLUTE: the expected set comes from +// `MemoryDependentAnalyzer::DepBetween` over a freshly built SyncIR, so it reads +// none of the sync allocator's decisions and there is no reference file. +// +// Two exemptions exist, and both are COUNTED rather than dropped, so that +// covered + uncovered + arm_excluded + arch_guaranteed == deps holds exactly. A +// predicate that quietly swallowed real dependencies would be indistinguishable +// from a correct one in any total-only check. + +// --------------------------------------------------------------------------- +// 1. Sound: every dependency ordered, on synchronization InsertSync produced. +// RUN: ptoas --pto-arch=a3 --enable-insert-sync --check-sync-self-coverage %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=COVERED +// COVERED: [sync-gate:g1-self] func=oracle_vadd compounds=4 deps=5 covered=5 uncovered=0 (samepipe=0 crosspipe=0) carried=0/0 arm_excluded=0 arch_guaranteed=0 pipe_self_ordered=0 unmappable=0 OK + +// 2. Not vacuous: the same kernel with no sync pass at all still has the same five +// dependencies, and none of them is ordered. `deps` is identical to the run +// above, which is what shows the expected set does not move with the candidate. +// RUN: not ptoas --pto-arch=a3 --check-sync-self-coverage %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=NOSYNC +// NOSYNC: [sync-gate:g1-self] func=oracle_vadd compounds=4 deps=5 covered=0 uncovered=5 (samepipe=0 crosspipe=5) carried=0/0 arm_excluded=0 arch_guaranteed=0 pipe_self_ordered=0 unmappable=0 + +// --------------------------------------------------------------------------- +// 3. THE A5 INTRA-PIPE GUARANTEE, and that it does not leak across arches. +// +// A5 orders two consecutive PIPE_V ops on one tile in hardware, and its backend +// rejects an explicit vector barrier for such a pair, so demanding one would make +// the gate unsatisfiable rather than strict. The rule is applied at exactly the +// condition the codegen uses, so the two cannot drift. +// +// On A5 the two same-pipe dependencies are credited to the target: +// RUN: ptoas --pto-arch=a5 --enable-insert-sync --check-sync-self-coverage %S/Inputs/oracle_gate_features.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=A5GUAR +// A5GUAR: [sync-gate:g1-self] func=same_pipe compounds=4 deps=5 covered=3 uncovered=0 (samepipe=0 crosspipe=0) carried=0/0 arm_excluded=0 arch_guaranteed=2 pipe_self_ordered=0 unmappable=0 OK + +// On A3 the same analysis of the same kernel credits NOTHING to the target -- +// `arch_guaranteed=0` is the property under test here, and it is what shows the +// A5 rule did not leak. A3 orders those dependencies with real sync instead. +// RUN: ptoas --pto-arch=a3 --enable-insert-sync --check-sync-self-coverage %S/Inputs/oracle_gate_features.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=A3NOGUAR +// A3NOGUAR: [sync-gate:g1-self] func=same_pipe compounds=4 deps=5 covered=5 uncovered=0 (samepipe=0 crosspipe=0) carried=0/0 arm_excluded=0 arch_guaranteed=0 pipe_self_ordered=0 unmappable=0 OK + +// --------------------------------------------------------------------------- +// 4. MUTUALLY EXCLUSIVE ARMS. Two ops in different arms of one `scf.if` can never +// both execute, so no ordering is missing between them. `scf.if` is the only +// mutually-exclusive-region op this recognises; the exclusion is counted so the +// identity above still balances (4 + 0 + 1 + 0 == 5). +// RUN: ptoas --pto-arch=a3 --enable-insert-sync --check-sync-self-coverage %S/Inputs/oracle_gate_features.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=ARMS +// ARMS: [sync-gate:g1-self] func=exclusive_arms compounds=4 deps=5 covered=4 uncovered=0 (samepipe=0 crosspipe=0) carried=0/0 arm_excluded=1 arch_guaranteed=0 pipe_self_ordered=0 unmappable=0 OK diff --git a/test/lit/pto/sync_gate_g1_sync_reachability.pto b/test/lit/pto/sync_gate_g1_sync_reachability.pto new file mode 100644 index 0000000000..4ea82c7a6f --- /dev/null +++ b/test/lit/pto/sync_gate_g1_sync_reachability.pto @@ -0,0 +1,40 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// A sync op earns coverage credit only where it is reached on every path that reaches +// the sink. Program order alone does not decide that, and neither does structural +// dominance -- the four functions below pin both boundaries. +// +// Every run is `not`: one of the four is genuinely uncovered, so the gate fails the +// pipeline. That also stops the three legal cases passing vacuously, because dropping +// the rule entirely makes the run succeed and `not` fail. + +// THE DEFECT. The barrier is on the `then` arm and both stores are outside the `if`, +// so the false path runs them back to back. Credit must NOT be given. +// RUN: not ptoas --pto-arch=a3 --check-sync-self-coverage %S/Inputs/oracle_sync_reachability.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=ARM +// ARM: [sync-gate:g1-self] func=sync_in_arm_outside_sink compounds=3 deps=3 covered=2 uncovered=1 (samepipe=1 crosspipe=0) carried=0/0 arm_excluded=0 arch_guaranteed=0 pipe_self_ordered=0 unmappable=0 +// ARM-NEXT: !! kind=uncovered-dependency waw anchor#1 (pto.tstore) -> anchor#2 (pto.tstore) PIPE_MTE3->PIPE_MTE3 + +// The same dependencies with the barrier hoisted out of the `if`. This one is ordered, +// which is what shows the rule keys on reachability and not on the presence of an `if`. +// RUN: not ptoas --pto-arch=a3 --check-sync-self-coverage %S/Inputs/oracle_sync_reachability.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=HOIST +// HOIST: [sync-gate:g1-self] func=sync_hoisted_covers compounds=3 deps=3 covered=3 uncovered=0 (samepipe=0 crosspipe=0) carried=0/0 arm_excluded=0 arch_guaranteed=0 pipe_self_ordered=0 unmappable=0 OK + +// LEGAL. Barrier and sink in the same arm: every path reaching the sink passes through +// the barrier. A rule that also required the sync to be reached whenever the SOURCE is +// would reject this, and it is the shape the emitter produces most. +// RUN: not ptoas --pto-arch=a3 --check-sync-self-coverage %S/Inputs/oracle_sync_reachability.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=ONEARM +// ONEARM: [sync-gate:g1-self] func=sync_and_sink_in_one_arm compounds=3 deps=3 covered=3 uncovered=0 (samepipe=0 crosspipe=0) carried=0/0 arm_excluded=0 arch_guaranteed=0 pipe_self_ordered=0 unmappable=0 OK + +// LEGAL, AND THE BOUND ON THE RULE. The barrier is inside a constant-trip-count loop +// and the sink follows the loop, so the barrier always runs. STRUCTURAL DOMINANCE +// disagrees -- a body op does not dominate an op after the loop -- so a dominance-keyed +// rule reports this uncovered. Enclosing loops are therefore not compared, and this is +// the line that fails if they ever are. +// RUN: not ptoas --pto-arch=a3 --check-sync-self-coverage %S/Inputs/oracle_sync_reachability.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=LOOP +// LOOP: [sync-gate:g1-self] func=sync_in_loop_outside_sink compounds=3 deps=4 covered=4 uncovered=0 (samepipe=0 crosspipe=0) carried=1/1 arm_excluded=0 arch_guaranteed=0 pipe_self_ordered=0 unmappable=0 OK diff --git a/test/lit/pto/sync_gate_g2_event_pair_guards.pto b/test/lit/pto/sync_gate_g2_event_pair_guards.pto new file mode 100644 index 0000000000..2d19cd3cc8 --- /dev/null +++ b/test/lit/pto/sync_gate_g2_event_pair_guards.pto @@ -0,0 +1,44 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// G2 pairs a set_flag with the wait_flag that closes its id. Program order alone +// cannot decide whether the two halves belong together: a pair placed on opposite +// arms of an `scf.if` balances the scan and its counts match, while no runtime path +// executes both. The pair must be EQUALLY GUARDED. +// +// Every run below is `not`: three of the five functions are illegal, so the gate +// fails the pipeline. That also makes the file self-guarding -- if the rule were +// removed entirely the run would succeed and `not` would fail, so the legal cases +// below cannot pass vacuously. + +// The set on one arm, the wait on the other. +// RUN: not ptoas --pto-arch=a3 --check-sync-interference %S/Inputs/oracle_event_pair_guards.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=ARMS +// ARMS: [sync-gate:g2] func=pair_split_across_arms view=ir records=2 errors=1 warnings=0 +// ARMS-NEXT: !! #1 pto.wait_flag kind=event-id-pair-guard-mismatch MTE2->MTE3 id=0 + +// A guarded set with an unguarded wait. The wait outlives its producer on the false +// path and the kernel hangs, which the arm case alone would not have caught. +// RUN: not ptoas --pto-arch=a3 --check-sync-interference %S/Inputs/oracle_event_pair_guards.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=GSET +// GSET: [sync-gate:g2] func=guarded_set_plain_wait view=ir records=2 errors=1 warnings=0 +// GSET-NEXT: !! #1 pto.wait_flag kind=event-id-pair-guard-mismatch MTE2->MTE3 id=0 + +// The mirror: an unguarded set whose wait is guarded leaks the id. +// RUN: not ptoas --pto-arch=a3 --check-sync-interference %S/Inputs/oracle_event_pair_guards.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=GWAIT +// GWAIT: [sync-gate:g2] func=plain_set_guarded_wait view=ir records=2 errors=1 warnings=0 +// GWAIT-NEXT: !! #1 pto.wait_flag kind=event-id-pair-guard-mismatch MTE2->MTE3 id=0 + +// LEGAL. Both halves under the same guard. +// RUN: not ptoas --pto-arch=a3 --check-sync-interference %S/Inputs/oracle_event_pair_guards.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=ONEARM +// ONEARM: [sync-gate:g2] func=pair_on_one_arm view=ir records=2 errors=0 warnings=0 OK + +// LEGAL, AND THE BOUND ON THE RULE. Set outside a loop, wait inside it: different +// blocks, same guards. This is the shape the emitter produces most, so a rule keyed +// on blocks rather than conditionals would reject working kernels -- this line is +// what fails if the comparison is ever tightened that way. +// RUN: not ptoas --pto-arch=a3 --check-sync-interference %S/Inputs/oracle_event_pair_guards.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=PRIME +// PRIME: [sync-gate:g2] func=set_outside_loop_wait_inside view=ir records=2 errors=0 warnings=0 OK diff --git a/test/lit/pto/sync_gate_g2_faults.pto b/test/lit/pto/sync_gate_g2_faults.pto new file mode 100644 index 0000000000..34ea7b351a --- /dev/null +++ b/test/lit/pto/sync_gate_g2_faults.pto @@ -0,0 +1,36 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// Two G2 shapes that only the SyncOperation view can judge, so they cannot be +// written as input IR: the emitted-IR view records `set_flag_dyn` with +// eventId = -1, and a compensation record's link to the hazard it primes exists +// only before codegen. They are appended to the gate's own record list, never to +// the IR, so they cannot reach codegen. +// +// Each pins the `event-id-nested` line specifically rather than an error total. +// The synthetic list is judged in isolation, so its lone sets also report as +// leaked ids -- true of the list, but not the property under test. + +// A compensation record shares its id with the hazard it primes BY DESIGN, so that +// pairing must not read as a collision. A collision with any OTHER hazard still +// must, and #902 is a different hazard from #900's owner. +// RUN: not ptoas --pto-arch=a3 --enable-insert-sync --check-sync-interference --check-sync-interference-inject-fault=comp-conflict %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=COMP +// COMP: [sync-gate:g2] func=oracle_vadd view=syncops records=2 +// COMP-NEXT: !! #902 set_event kind=event-id-nested MTE2->V id=3 live_since=#900 + +// The scan must check EVERY id a hazard holds. These two records differ on their +// first id and collide on their second, so a first-id-only scan finds nothing. +// RUN: not ptoas --pto-arch=a3 --enable-insert-sync --check-sync-interference --check-sync-interference-inject-fault=multi-id-conflict %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=MULTI +// MULTI: [sync-gate:g2] func=oracle_vadd view=syncops records=2 +// MULTI-NEXT: !! #912 set_event kind=event-id-nested MTE2->V id=1 live_since=#910 + +// A selector naming no shape must fail loudly, for the same reason G3's does: +// injecting nothing would leave the gate with nothing to find, so the two runs +// above would pass while proving nothing. +// RUN: not ptoas --pto-arch=a3 --enable-insert-sync --check-sync-interference --check-sync-interference-inject-fault=typo-not-a-shape %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=BADSEL +// BADSEL: error: unknown --check-sync-interference-inject-fault selector 'typo-not-a-shape'; expected comp-conflict or multi-id-conflict diff --git a/test/lit/pto/sync_gate_g2_legal.pto b/test/lit/pto/sync_gate_g2_legal.pto new file mode 100644 index 0000000000..043d0d4aac --- /dev/null +++ b/test/lit/pto/sync_gate_g2_legal.pto @@ -0,0 +1,20 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// G2 passes synchronization the existing allocator produced, including the case +// that would be a false positive under the wrong key: this kernel uses EVENT_ID0 +// in BOTH MTE2->V and V->MTE3 at once, which is legal because event flags are +// per-direction disjoint hardware. Keying on the id alone would report it. + +// RUN: ptoas --pto-arch=a3 --enable-insert-sync --check-sync-interference %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=LEGAL +// LEGAL: [sync-gate:g2] func=oracle_vadd view=ir records=5 errors=0 warnings=0 OK + +// The two directions really do share id 0 here, so the check above is not vacuous. +// RUN: ptoas --pto-arch=a3 --enable-insert-sync --emit-pto-ir %S/Inputs/oracle_vadd.pto | FileCheck %s --check-prefix=SHARED +// SHARED-DAG: pto.set_flag[, , ] +// SHARED-DAG: pto.set_flag[, , ] diff --git a/test/lit/pto/sync_gate_g3_faults.pto b/test/lit/pto/sync_gate_g3_faults.pto new file mode 100644 index 0000000000..53ddc4c657 --- /dev/null +++ b/test/lit/pto/sync_gate_g3_faults.pto @@ -0,0 +1,36 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// G3 rejects ids the target forbids. None of the three classes below can be written +// as input IR: `pto.set_flag` spells its id as an EVENT_ID0..EVENT_ID7 enum, so 9 and +// 14 are unrepresentable, and `pto.get_buf`'s buf_id is verified into [0, 31], so 64 +// is too. Each fault is appended to the gate's own record list after extraction, +// never to the IR, so it cannot reach codegen. `--check-sync-ids-inject-fault` exists +// for that and is hidden. + +// An event id past the end of the intra-core pool. +// RUN: not ptoas --pto-arch=a3 --check-sync-ids --check-sync-ids-inject-fault=event-oor %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=OOR +// OOR: [sync-gate:g3] func=oracle_vadd view=syncops records=1 violations=1 +// OOR-NEXT: !! #0 set_event kind=event-id-out-of-range id=9 : intra-core event pool for V->MTE3 is [0, 8) + +// An id that `sync_block_all` owns while one is live. +// RUN: not ptoas --pto-arch=a3 --check-sync-ids --check-sync-ids-inject-fault=block-all-stomp %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=STOMP +// STOMP: [sync-gate:g3] func=oracle_vadd view=syncops records=2 violations=1 +// STOMP-NEXT: !! #1 sync_block_set kind=event-id-reserved-for-block-all id=14 : ids 14/15 belong to sync_block_all while one is live + +// A buffer id past the hardware field. Checked on A5, where buffer-id sync exists. +// RUN: not ptoas --pto-arch=a5 --check-sync-ids --check-sync-ids-inject-fault=bufid-oor %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=BUFOOR +// BUFOOR: [sync-gate:g3] func=oracle_vadd view=ir records=1 violations=1 +// BUFOOR-NEXT: !! #0 pto.get_buf kind=buf-id-out-of-range id=64 : hardware buf_id field is [0, 32) + +// A SELECTOR THAT NAMES NO FAULT MUST FAIL LOUDLY. Injecting nothing would leave +// the gate with nothing to find, so it would report clean and the three runs above +// would pass while proving nothing -- a misspelling would vacuously satisfy the +// tests that exist to show the gate is not vacuous. +// RUN: not ptoas --pto-arch=a3 --check-sync-ids --check-sync-ids-inject-fault=typo-not-a-fault %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=BADSEL +// BADSEL: error: unknown --check-sync-ids-inject-fault selector 'typo-not-a-fault'; expected event-oor, block-all-stomp or bufid-oor diff --git a/test/lit/pto/sync_gate_g3_legal.pto b/test/lit/pto/sync_gate_g3_legal.pto new file mode 100644 index 0000000000..5df3045b8d --- /dev/null +++ b/test/lit/pto/sync_gate_g3_legal.pto @@ -0,0 +1,19 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// G3 passes synchronization the existing allocator produced. Paired with +// sync_gate_g3_faults.pto: a gate that only ever passed would clear this file too. + +// RUN: ptoas --pto-arch=a3 --enable-insert-sync --check-sync-ids %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=A3 +// A3: [sync-gate:g3] func=oracle_vadd view=ir records=5 violations=0 OK + +// The same kernel on A5. This kernel emits no buffer-id op, so the arch-capacity +// rule is not exercised here; what the line pins is that changing the arch does not +// change the verdict on event ids. +// RUN: ptoas --pto-arch=a5 --enable-insert-sync --check-sync-ids %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=A5 +// A5: [sync-gate:g3] func=oracle_vadd view=ir records=5 violations=0 OK diff --git a/test/lit/pto/sync_gate_g4_count_floor.pto b/test/lit/pto/sync_gate_g4_count_floor.pto new file mode 100644 index 0000000000..c7d9521d6c --- /dev/null +++ b/test/lit/pto/sync_gate_g4_count_floor.pto @@ -0,0 +1,39 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// G4 compares one compilation's sync-op counts against a reference compilation of +// the same function, per class rather than in total. + +// --------------------------------------------------------------------------- +// 1. A run compared against itself must be clean, and the profile it writes is +// what every case below is measured against. +// RUN: ptoas --pto-arch=a3 --enable-insert-sync --dump-sync-counts %S/Inputs/oracle_vadd.pto 2> %t.ref > /dev/null +// RUN: FileCheck %s --check-prefix=PROFILE --input-file=%t.ref +// PROFILE: [sync-count] func=oracle_vadd set=2 wait=2 get=0 rls=0 barrier=1 barrier_all=1 total=5 + +// RUN: ptoas --pto-arch=a3 --enable-insert-sync --check-sync-count-floor=%t.ref %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=SELF +// SELF: [sync-gate:g4] func=oracle_vadd candidate_total=5 reference_total=5 violations=0 OK + +// --------------------------------------------------------------------------- +// 2. WHY THE FLOOR IS PER CLASS AND NOT A TOTAL. The barrier-all mode replaces the +// two directed pairs with five full-core barriers. The TOTAL IS IDENTICAL at 5, +// so a scalar floor would wave through a fully serializing allocation; only the +// per-class rules catch it. +// RUN: not ptoas --pto-arch=a3 --enable-inject-barrier-all-sync --check-sync-count-floor=%t.ref %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=DEGEN +// DEGEN: [sync-gate:g4] func=oracle_vadd candidate_total=5 reference_total=5 violations=2 +// DEGEN: candidate: [sync-count] func=oracle_vadd set=0 wait=0 get=0 rls=0 barrier=5 barrier_all=5 total=5 +// DEGEN: !! kind=barrier-count-increased candidate=5 reference=1 +// DEGEN: !! kind=barrier-all-count-increased candidate=5 reference=1 + +// --------------------------------------------------------------------------- +// 3. A function with no entry in the reference is a violation, not a pass. A gate +// that silently skips what it cannot find is not a gate. +// RUN: echo "[sync-count] func=some_other_kernel set=0 wait=0 get=0 rls=0 barrier=0 barrier_all=0 total=0" > %t.wrong +// RUN: not ptoas --pto-arch=a3 --enable-insert-sync --check-sync-count-floor=%t.wrong %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=MISSING +// MISSING: [sync-gate:g4] func=oracle_vadd violations=1 +// MISSING-NEXT: !! kind=missing-reference-profile diff --git a/test/lit/pto/sync_mechanism_field.pto b/test/lit/pto/sync_mechanism_field.pto new file mode 100644 index 0000000000..e802db7e9d --- /dev/null +++ b/test/lit/pto/sync_mechanism_field.pto @@ -0,0 +1,96 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===----------------------------------------------------------------------===// +// THE `mechanism` FIELD ON SyncOperation. +// +// `mechanism` is a DIFFERENT AXIS from `TYPE`. TYPE is the op shape -- a set, a wait, +// a barrier -- and the analysis decides it. MECHANISM is the resource class backing +// it, and therefore where the sync may be placed: +// +// event / barrier -> carried at a SyncIR position, hoistable by MoveSyncState +// bufid -> MUST be op-anchored, bracketing the access +// +// Buffer routing is decided per BUFFER by `routeBuffers`, so on a kernel it declines +// no hazard carries `bufid` at all. The hidden `--unified-sync-force-mechanism` flag +// stamps it anyway, so the value and its plumbing are exercised rather than shipped +// as an enum case that first executes in anger. +//===----------------------------------------------------------------------===// + +// --------------------------------------------------------------------------- +// 1. DEFAULT -- mechanism is DERIVED FROM TYPE, not defaulted flat to EVENT. +// +// This is why the default is derived rather than assigned. Under a flat +// "default = EVENT", record #2 -- a PIPE_ALL barrier -- would report `mech=event`, +// so every barrier would claim an event id it does not hold. Deriving it means no +// construction site has to remember to set the field, and a barrier cannot pose as +// an event. +// RUN: ptoas --pto-arch=a3 --enable-unified-sync --enable-unified-sync-debug %S/Inputs/oracle_vadd.pto 2>&1 | FileCheck %s --check-prefix=DEFAULT +// DEFAULT: sync-extract:syncops{{.*}} records=5 +// DEFAULT-NEXT: #0 set_flag mech=event src=MTE2 dst=V +// DEFAULT-NEXT: #0 wait_flag mech=event src=MTE2 dst=V +// DEFAULT-NEXT: #1 set_flag mech=event src=V dst=MTE3 +// DEFAULT-NEXT: #1 wait_flag mech=event src=V dst=MTE3 +// DEFAULT-NEXT: #2 pipe_barrier mech=barrier src=ALL dst=ALL + +// --------------------------------------------------------------------------- +// 2. STAMPED BUFID -- both halves of the pair, and only that hazard. +// +// Two properties are pinned, and both are load-bearing: +// +// (a) BOTH halves of hazard #0 carry `bufid`. A set routed to a buffer token whose +// wait stayed on an event is not a pair -- it is two unrelated syncs that +// cannot discharge each other. Mechanism is therefore a per-HAZARD property, +// stamped on set and wait together, never per SyncOperation. +// +// (b) The kernel is MIXED: #0 on bufid, #1 still on event, the tail still a +// barrier. Three mechanisms coexisting in one kernel is the state buffer +// routing produces, so it must be representable. +// +// THE RUN REFUSES, AND THE REFUSAL IS THE POINT. Stamping the mechanism makes +// `colorEventIds` skip the hazard, but nothing routed a buffer for it -- routing is a +// per-buffer decision this flag does not make -- so the pair reaches codegen with no +// id. Without the guard it would vanish silently, with ptoas exiting 0 and every gate +// clean, because a dropped ordering leaves nothing for a gate to object to. The +// records still print, so the plumbing under test stays observable; what fails is the +// emission. +// RUN: not ptoas --pto-arch=a5 --enable-unified-sync --enable-unified-sync-debug --unified-sync-force-mechanism=bufid %S/Inputs/oracle_vadd.pto 2>&1 | FileCheck %s --check-prefix=BUFID +// BUFID: sync-extract:syncops{{.*}} records=5 +// BUFID-NEXT: #0 set_flag mech=bufid src=MTE2 dst=V +// BUFID-NEXT: #0 wait_flag mech=bufid src=MTE2 dst=V +// BUFID-NEXT: #1 set_flag mech=event src=V dst=MTE3 +// BUFID-NEXT: #1 wait_flag mech=event src=V dst=MTE3 +// BUFID-NEXT: #2 pipe_barrier mech=barrier src=ALL dst=ALL +// BUFID: error: sync codegen +// BUFID-SAME: no mechanism was realised for it + +// --------------------------------------------------------------------------- +// 3. SPILL -- the resource-exhaustion downgrade drops TYPE and MECHANISM together. +// +// `SetPipeAll` is the overflow path: it rewrites the type to PIPE_BARRIER, and it +// must drop the mechanism with it. Otherwise these two records read +// `pipe_barrier mech=bufid` -- a spilled barrier still claiming a buffer token it no +// longer holds, which codegen would try to back with a get/release pair that was +// never allocated. +// RUN: ptoas --pto-arch=a5 --enable-unified-sync --enable-unified-sync-debug --unified-sync-force-mechanism=bufid-spill %S/Inputs/oracle_vadd.pto 2>&1 | FileCheck %s --check-prefix=SPILL +// SPILL: sync-extract:syncops{{.*}} records=5 +// SPILL-NEXT: #0 pipe_barrier mech=barrier src=ALL dst=ALL +// SPILL-NEXT: #0 pipe_barrier mech=barrier src=ALL dst=ALL +// SPILL-NEXT: #1 set_flag mech=event src=V dst=MTE3 +// SPILL-NEXT: #1 wait_flag mech=event src=V dst=MTE3 +// SPILL-NEXT: #2 pipe_barrier mech=barrier src=ALL dst=ALL + +// --------------------------------------------------------------------------- +// 4. WHY THERE IS NO CHECK-NOT OVER THE EMITTED CODE HERE. +// +// A `CHECK-NOT: set_flag` over the emitted C++ would be a false pin: ptoas emits the +// `ptoas_auto_sync_tail` helper DEFINITION into every output regardless of sync mode, +// and its body contains a literal `set_flag(PIPE_MTE3, PIPE_S, EVENT_ID0)`. Such a +// check fails even on a kernel that emits no sync at all. Sync-op counts must be +// taken over the KERNEL BODY, which is what the oracle's IR view does, never over the +// whole emitted file. diff --git a/test/lit/pto/sync_oracle_self_validation.pto b/test/lit/pto/sync_oracle_self_validation.pto new file mode 100644 index 0000000000..33fffa6e9f --- /dev/null +++ b/test/lit/pto/sync_oracle_self_validation.pto @@ -0,0 +1,75 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// Proves the three gates are neither unsound nor vacuous, which is the only basis +// on which a green result from them means anything. +// +// Soundness: all three pass on synchronization InsertSync itself produced. +// Non-vacuity: each of the three fault classes below is caught by exactly ONE gate +// and is SILENT on the other two. Delete any one gate and a fault class escapes. +// +// fault class G1-self G2 G3 +// --------------------------------------------------------- +// nothing ordered at all CATCH pass pass +// two overlapping uses of one id pass CATCH pass +// an id the target forbids pass pass CATCH +// +// The inputs are hand-written modules rather than compiler output, so no gate's +// verdict depends on another pass continuing to behave a particular way. + +// --------------------------------------------------------------------------- +// ROW 0 -- SOUNDNESS. All three green on InsertSync's own output. +// RUN: ptoas --pto-arch=a3 --enable-insert-sync --check-sync-self-coverage %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=GOOD1 +// GOOD1: [sync-gate:g1-self] func=oracle_vadd compounds=4 deps=5 covered=5 uncovered=0 (samepipe=0 crosspipe=0) carried=0/0 arm_excluded=0 arch_guaranteed=0 pipe_self_ordered=0 unmappable=0 OK + +// RUN: ptoas --pto-arch=a3 --enable-insert-sync --check-sync-interference %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=GOOD2 +// GOOD2: [sync-gate:g2] func=oracle_vadd view=ir records=5 errors=0 warnings=0 OK + +// RUN: ptoas --pto-arch=a3 --enable-insert-sync --check-sync-ids %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=GOOD3 +// GOOD3: [sync-gate:g3] func=oracle_vadd view=ir records=5 violations=0 OK + +// --------------------------------------------------------------------------- +// ROW 1 -- NOTHING ORDERED AT ALL. Only G1-self can see this. +// +// The fault is the ABSENCE of a sync pass, so the source is the whole truth and +// this row cannot expire the way a stub-based injector would: it depends on no +// pass's behaviour. G2 and G3 are silent because they judge the ids that ARE +// present, and there are none -- which is exactly why a green G2/G3 is not a +// correctness result on its own. +// RUN: not ptoas --pto-arch=a3 --check-sync-self-coverage %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=EMPTY1 +// EMPTY1: [sync-gate:g1-self] func=oracle_vadd compounds=4 deps=5 covered=0 uncovered=5 (samepipe=0 crosspipe=5) carried=0/0 arm_excluded=0 arch_guaranteed=0 pipe_self_ordered=0 unmappable=0 + +// RUN: ptoas --pto-arch=a3 --check-sync-interference %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=EMPTY2 +// EMPTY2: [sync-gate:g2] func=oracle_vadd view=ir records=0 errors=0 warnings=0 OK + +// RUN: ptoas --pto-arch=a3 --check-sync-ids %S/Inputs/oracle_vadd.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=EMPTY3 +// EMPTY3: [sync-gate:g3] func=oracle_vadd view=ir records=0 violations=0 OK + +// --------------------------------------------------------------------------- +// ROW 2 -- TWO OVERLAPPING USES OF ONE ID. Only G2 can see this. +// +// One (src,dst,id) is acquired twice before either release. The op count is +// unchanged and every ordering is still established, so neither a coverage gate +// nor a count floor can distinguish it from correct code. +// RUN: not ptoas --pto-arch=a3 --check-sync-interference %S/Inputs/oracle_selfval_nested_id.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=NEST1 +// NEST1: [sync-gate:g2] func=selfval_nested_id view=ir records=4 errors=1 warnings=0 + +// RUN: ptoas --pto-arch=a3 --check-sync-ids %S/Inputs/oracle_selfval_nested_id.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=NEST2 +// NEST2: [sync-gate:g3] func=selfval_nested_id view=ir records=4 violations=0 OK + +// --------------------------------------------------------------------------- +// ROW 3 -- AN ID THE TARGET FORBIDS. Only G3 can see this. +// +// A direction that reserves the tail of its pool, used with a reserved id. The +// intervals are disjoint and every dependency is ordered, so G2 and G1-self are +// both correct to stay quiet. +// RUN: not ptoas --pto-arch=a3 --check-sync-ids %S/Inputs/oracle_selfval_illegal_id.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=ILL1 +// ILL1: [sync-gate:g3] func=selfval_illegal_id view=ir records=2 violations=2 + +// RUN: ptoas --pto-arch=a3 --check-sync-interference %S/Inputs/oracle_selfval_illegal_id.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=ILL2 +// ILL2: [sync-gate:g2] func=selfval_illegal_id view=ir records=2 errors=0 warnings=0 OK diff --git a/test/lit/pto/sync_overflow_id_gap.pto b/test/lit/pto/sync_overflow_id_gap.pto new file mode 100644 index 0000000000..e5c1a8c174 --- /dev/null +++ b/test/lit/pto/sync_overflow_id_gap.pto @@ -0,0 +1,129 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===----------------------------------------------------------------------===// +// BUFFER-ID ROUTING ABSORBS AN EVENT-POOL OVERFLOW. +// +// This kernel's tiles are written back AND overflow the 8-id event pool, so the +// router sends them to the token protocol and the colourer never overflows -- +// nothing reaches the barrier spill. One tile, the output, is NOT written back, so +// it stays on events: that is why a single kernel exercises both sides of the +// routing predicate and the counts below discriminate rather than merely pass. +// +// The mirror case -- overflow that is NOT written back, so it must spill -- is +// pinned by `forced_overflow_n12` in sync_forced_overflow.pto. Neither file covers +// the other. +// +// THE FORBIDDEN IDS ARE THE POINT. `2147483647` is the unset sentinel and +// `event-id-nested id=0` is an id nobody handed out; either one means a hazard +// reached codegen with no mechanism realised and its ordering was silently dropped. +// Do not relax these checks: they forbid ids the hardware was never told about. +//===----------------------------------------------------------------------===// + +// Routing takes all 12 written-back overflowing buffers, and the colourer is left +// with only the 2 hazards on the un-routed output tile -- `overflow=0`, so the +// spill never fires here. +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-unified-sync --enable-unified-sync-debug %s 2>&1 >/dev/null | FileCheck %s --check-prefix=ROUTED +// ROUTED: [unified-sync routing] func=overflow_id_gap buffer routing: considered=13 routed=12 hazards_covered=24 skipped(no_overflow=1 not_written_back=0 K=0:0) not_expressible=0 split=0 OK +// ROUTED: [unified-sync coloring] interval coloring: assigned=2 overflow=0 spilled_to_barrier=0 hidden_reserved=0 reused=0 + +// The emission actually happened, and every acquire has its release. +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-unified-sync --enable-unified-sync-debug %s 2>&1 >/dev/null | FileCheck %s --check-prefix=EMITTED +// EMITTED: [unified-sync bufid] func=overflow_id_gap bufid emission: routed_buffers=12 routed_hazards=24 anchor_ops=24 get=24 rls=24 physical_ids=13 nesting=OK emit=OK bracketed_hazards=24 unbracketed_hazards=0 suppressed_event_ops=72 leaked_event_ops=0 ALL-OR-NOTHING-OK + +// G2 -- the gate that caught the original defect -- is clean, and B1-B7 are being +// applied to OUR buffer-ID allocation for the first time. No B5 (single-pipe id), +// and zero B6 warnings. +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-unified-sync --enable-unified-sync-debug --check-sync-interference %s 2>&1 >/dev/null | FileCheck %s --check-prefix=G2CLEAN +// G2CLEAN: [sync-gate:g2] func=overflow_id_gap view=ir records={{[0-9]+}} errors=0 warnings=0 OK + +// G3 device-id legality, on both views. Buffer ids must be within K=32. +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-unified-sync --enable-unified-sync-debug --check-sync-ids %s 2>&1 >/dev/null | FileCheck %s --check-prefix=G3 +// G3: [sync-gate:g3] func=overflow_id_gap view=ir records={{[0-9]+}} violations=0 OK + +// G1-self: the token protocol must actually COVER the orderings it replaced -- +// rls-before-get on the same id, which G1-self models directly. All 90 loop-carried +// dependencies stay covered with the event pairs gone. +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-unified-sync --enable-unified-sync-debug --check-sync-self-coverage %s 2>&1 >/dev/null | FileCheck %s --check-prefix=SELF +// SELF: [sync-gate:g1-self] func=overflow_id_gap compounds=25 deps=205 covered=139 uncovered=0 (samepipe=0 crosspipe=0) carried=115/115 arm_excluded=0 arch_guaranteed=66 pipe_self_ordered=0 unmappable=0 OK + +// THE EMITTED FORM. get_buf/rls_buf present, sentinel absent, and the whole-core +// serialisation gone -- exactly one PIPE_ALL survives (the auto-sync tail). +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-unified-sync --enable-unified-sync-debug %s -o - 2>/dev/null | FileCheck %s --check-prefix=EMIT +// EMIT: get_buf( +// EMIT: rls_buf( +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-unified-sync --enable-unified-sync-debug %s -o - 2>/dev/null | FileCheck %s --check-prefix=NOSENTINEL +// NOSENTINEL-NOT: 2147483647 + +module attributes {pto.target_arch = "a5"} { + func.func @overflow_id_gap(%src: !pto.ptr, %dst: !pto.ptr) attributes {pto.kernel_kind = #pto.kernel_kind} { + %c0 = arith.constant 0 : index + %c1 = arith.constant 1 : index + %c32 = arith.constant 32 : index + %c4 = arith.constant 4 : index + %a0 = arith.constant 0 : i64 + %a1 = arith.constant 2048 : i64 + %a2 = arith.constant 4096 : i64 + %a3 = arith.constant 6144 : i64 + %a4 = arith.constant 8192 : i64 + %a5 = arith.constant 10240 : i64 + %a6 = arith.constant 12288 : i64 + %a7 = arith.constant 14336 : i64 + %a8 = arith.constant 16384 : i64 + %a9 = arith.constant 18432 : i64 + %a10 = arith.constant 20480 : i64 + %a11 = arith.constant 22528 : i64 + %aout = arith.constant 24576 : i64 + %sv = pto.make_tensor_view %src, shape = [%c32, %c32], strides = [%c32, %c1] : !pto.tensor_view + %dv = pto.make_tensor_view %dst, shape = [%c32, %c32], strides = [%c32, %c1] : !pto.tensor_view + %ps = pto.partition_view %sv, offsets = [%c0, %c0], sizes = [%c32, %c32] : !pto.tensor_view -> !pto.partition_tensor_view<32x32xf16> + %pd = pto.partition_view %dv, offsets = [%c0, %c0], sizes = [%c32, %c32] : !pto.tensor_view -> !pto.partition_tensor_view<32x32xf16> + scf.for %it = %c0 to %c4 step %c1 { + %t0 = pto.alloc_tile addr = %a0 : !pto.tile_buf + %t1 = pto.alloc_tile addr = %a1 : !pto.tile_buf + %t2 = pto.alloc_tile addr = %a2 : !pto.tile_buf + %t3 = pto.alloc_tile addr = %a3 : !pto.tile_buf + %t4 = pto.alloc_tile addr = %a4 : !pto.tile_buf + %t5 = pto.alloc_tile addr = %a5 : !pto.tile_buf + %t6 = pto.alloc_tile addr = %a6 : !pto.tile_buf + %t7 = pto.alloc_tile addr = %a7 : !pto.tile_buf + %t8 = pto.alloc_tile addr = %a8 : !pto.tile_buf + %t9 = pto.alloc_tile addr = %a9 : !pto.tile_buf + %t10 = pto.alloc_tile addr = %a10 : !pto.tile_buf + %t11 = pto.alloc_tile addr = %a11 : !pto.tile_buf + %tout = pto.alloc_tile addr = %aout : !pto.tile_buf + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t0 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t1 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t2 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t3 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t4 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t5 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t6 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t7 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t8 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t9 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t10 : !pto.tile_buf) + pto.tload ins(%ps : !pto.partition_tensor_view<32x32xf16>) outs(%t11 : !pto.tile_buf) + pto.tcvt ins(%t0{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t1{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t2{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t3{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t4{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t5{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t6{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t7{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t8{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t9{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t10{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tcvt ins(%t11{rmode = #pto} : !pto.tile_buf) outs(%tout : !pto.tile_buf) + pto.tstore ins(%tout : !pto.tile_buf) outs(%pd : !pto.partition_tensor_view<32x32xf16>) + scf.yield + } + return + } +} diff --git a/test/lit/pto/sync_routed_token_ids.pto b/test/lit/pto/sync_routed_token_ids.pto new file mode 100644 index 0000000000..85998d9b32 --- /dev/null +++ b/test/lit/pto/sync_routed_token_ids.pto @@ -0,0 +1,57 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===----------------------------------------------------------------------===// +// A ROUTED BUFFER'S TWO ENDS MUST CARRY THE SAME TOKEN ID. +// +// A buffer-id token orders a producer before a consumer only if BOTH acquire the same id. +// The unified allocator decides which buffers to route by logical id, and a later stage +// that rewrote logical ids would break that correspondence in one of two ways: a bracket +// dropped, so nothing enforces the ordering; or the two ends left holding different ids, +// so the token orders nothing while the event ops it displaced are already gone. +// +// That is the shape `BufidSyncAnalysis::optimizeSamePipeMerge` produces -- it rewrites +// BufSyncOperation::logicId to a survivor id. The unified pass no longer calls it, because +// the hazard-driven rework replaced insertSyncOperations() and the merge together, so a +// routed set captured earlier can no longer go stale behind a rewrite. +// +// WHAT THIS TEST DOES AND DOES NOT PROVE. It pins the invariant: both ends of every routed +// hazard hold the same token id, and the routed direction retains no event ops. The +// existing overflow fixtures pin the emission COUNTS only, so a change that preserved +// counts while permuting ids would have passed all of them and this catches that. +// +// It is NOT a proven guard against reintroducing the merge. Adding that call back on this +// kernel was measured to change nothing -- get, rls and physical_ids all identical -- so +// this input does not distinguish the two. A guard against that specific regression would +// need an input the merge actually rewrites, and none is known. +//===----------------------------------------------------------------------===// + +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-unified-sync %S/Inputs/routed_token_ids.pto 2>/dev/null | FileCheck %s --check-prefix=TOKENS +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-unified-sync %S/Inputs/routed_token_ids.pto 2>/dev/null | FileCheck %s --check-prefix=NOEVENT +// RUN: ptoas --pto-arch=a5 --pto-level=level3 --enable-unified-sync --enable-unified-sync-debug %S/Inputs/routed_token_ids.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=COUNTS + +// The producing end brackets the load, and the id is captured rather than written down: +// what matters is that the SAME id comes back on the consuming end, not which one it is. +// TOKENS: get_buf(PIPE_MTE2, [[ID:[0-9]+]], 0); +// TOKENS-NEXT: TLOAD( +// TOKENS-NEXT: rls_buf(PIPE_MTE2, [[ID]], 0); + +// The consuming end, further down the function after the other eleven loads, must hold +// that same id. If a rewrite renumbered one side, this is the line that fails. +// TOKENS: get_buf(PIPE_V, [[ID]], 0); +// TOKENS-NEXT: TCVT( +// TOKENS-NEXT: rls_buf(PIPE_V, [[ID]], 0); + +// The other half of all-or-nothing: the routed direction keeps NO event ops. The kernel +// still has MTE3<->V pairs for the accumulator, which does not route, so this asserts +// absence for the routed direction specifically rather than for events in general. +// NOEVENT-NOT: set_flag(PIPE_MTE2, PIPE_V +// NOEVENT-NOT: wait_flag(PIPE_MTE2, PIPE_V + +// Twelve routed buffers, both ends of all twenty-four hazards bracketed, nothing leaked. +// COUNTS: routed_buffers=12 routed_hazards=24 anchor_ops=24 get=24 rls=24 physical_ids=13 nesting=OK emit=OK bracketed_hazards=24 unbracketed_hazards=0 suppressed_event_ops=72 leaked_event_ops=0 ALL-OR-NOTHING-OK diff --git a/test/lit/pto/sync_unified_compensation_g2.pto b/test/lit/pto/sync_unified_compensation_g2.pto new file mode 100644 index 0000000000..08f3eda0b0 --- /dev/null +++ b/test/lit/pto/sync_unified_compensation_g2.pto @@ -0,0 +1,89 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +// G2's syncops view must not read a rotating hazard's own compensation pair as a +// competing hazard -- and must still catch a real collision. +// +// THE OVER-APPROXIMATION THIS GUARDS AGAINST. A d>1 hazard is ONE syncops record +// declaring all d ids, and the compensation pair that primes and drains it declares the +// SAME ids over the enclosing interval -- SetEventIds stamps set, wait, head and tail +// alike. A linear scan therefore sees set;set;wait;wait on one (direction, id) and +// reports `event-id-nested`, failing a kernel the incumbent compiles, with zero sync +// emitted. The emitted shape is in fact the incumbent's: d primes, a dyn in-body pair, +// d drains. +// +// THE EXEMPTION IS OWNER-SCOPED, AND THAT IS THE PART THAT MATTERS. A compensation +// record reports the hazard it primes (`compensationOf`); everything else reports +// itself, and two opens conflict unless they share a non-sentinel owner. Excusing every +// compensation record instead would make G2 look fixed while dropping rotation coverage +// -- and syncops is the ONLY view that checks rotating ids, because the IR view's +// set_flag_dyn / wait_flag_dyn carry eventId = -1 and never enter the id check. +// +// THE NEGATIVE SIDE LIVES IN sync_gate_g2_faults.pto, which injects a compensation +// record colliding with an unrelated hazard, and a pair colliding only on a non-first +// id, directly into the gate's record list. Those pin that the scan still reports a +// real collision and that it checks every id rather than the first. This file pins the +// positive side: that the exemption does not fire on a hazard's own compensation. + +// --------------------------------------------------------------------------- +// 1. The kernel that used to fail now compiles, WITH sync emitted (not zero). +// RUN: ptoas --pto-arch=a3 --enable-unified-sync --enable-unified-sync-debug --emit-pto-ir %S/multi_tile_affine_disjoint_slots.pto | FileCheck %s --check-prefix=EMIT +// The order is the property, and it is the incumbent's: d=2 static primes BEFORE the +// loop, the rotating dyn pair INSIDE it, d=2 static drains AFTER. A prime that landed +// after the loop, or a missing one, is the iteration-1 deadlock. +// EMIT: pto.set_flag[, , ] +// EMIT-NEXT: pto.set_flag[, , ] +// EMIT-NEXT: pto.set_flag[, , ] +// EMIT-NEXT: pto.set_flag[, , ] +// EMIT: pto.wait_flag_dyn[, +// EMIT: pto.set_flag_dyn[, +// EMIT: pto.wait_flag_dyn[, +// EMIT: pto.set_flag_dyn[, +// EMIT: pto.wait_flag[, , ] +// EMIT-NEXT: pto.wait_flag[, , ] +// EMIT-NEXT: pto.wait_flag[, , ] +// EMIT-NEXT: pto.wait_flag[, , ] + +// 2. G2 on the syncops view is clean on it -- this is the pairing that used to error. +// RUN: ptoas --pto-arch=a3 --enable-unified-sync --enable-unified-sync-debug %S/multi_tile_affine_disjoint_slots.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=G2OK +// G2OK: [sync-gate:g2] func=prefetch_disjoint_slots view=syncops records=10 errors=0 warnings=0 OK + +// 3. fe != inv is now COMPARED, not just printed. It was reporting a number nobody +// read; it would have flagged this kernel the first time it ran. A WARNING, not an +// error: a divergence is not by itself wrong, and erroring would refuse six +// multi_tile kernels that compile today. +// RUN: ptoas --pto-arch=a3 --enable-unified-sync --enable-unified-sync-debug %S/multi_tile_affine_disjoint_slots.pto 2>&1 >/dev/null | FileCheck %s --check-prefix=DIVERGE +// DIVERGE: !! predicate-divergence func=prefetch_disjoint_slots fe=2 inv=1 +// DIVERGE-SAME: whose in-body set precedes its wait + +// --------------------------------------------------------------------------- +// 4. G2 IS CLEAN UNDER THIS ALLOCATOR ON A STRAIGHT-LINE KERNEL. sync_gate_g2_legal.pto +// pins the same property for the incumbent; this pins it for the unified path, so a +// gate that only ever ran green on the incumbent cannot pass for the wrong reason. +// RUN: ptoas --pto-arch=a3 --enable-unified-sync --enable-unified-sync-debug %s 2>&1 >/dev/null | FileCheck %s --check-prefix=CLEAN +// CLEAN: [sync-gate:g2] func=vadd_g2_control view=syncops records=5 errors=0 warnings=0 OK + + +module { + func.func @vadd_g2_control(%a: !pto.partition_tensor_view<1x32xf32>, + %b: !pto.partition_tensor_view<1x32xf32>, + %c: !pto.partition_tensor_view<1x32xf32>) { + %ta = pto.alloc_tile : !pto.tile_buf + %tb = pto.alloc_tile : !pto.tile_buf + pto.tload ins(%a : !pto.partition_tensor_view<1x32xf32>) + outs(%ta : !pto.tile_buf) + pto.tload ins(%b : !pto.partition_tensor_view<1x32xf32>) + outs(%tb : !pto.tile_buf) + pto.tadd ins(%ta, %tb : !pto.tile_buf, + !pto.tile_buf) + outs(%ta : !pto.tile_buf) + pto.tstore ins(%ta : !pto.tile_buf) + outs(%c : !pto.partition_tensor_view<1x32xf32>) + return + } +} diff --git a/test/lit/pto/sync_unified_macro_pinning.pto b/test/lit/pto/sync_unified_macro_pinning.pto new file mode 100644 index 0000000000..f1fc80cca5 --- /dev/null +++ b/test/lit/pto/sync_unified_macro_pinning.pto @@ -0,0 +1,78 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. +// A macro op's library-internal event ids are RESERVED, so the allocator cannot +// hand one to a compiler hazard. +// +// pto.mgather lowers to a pto-isa library call that performs its own +// PtoSetWaitFlag() internally, defaulted to EVENT_ID0. No op in the IR +// names that use, so nothing downstream can see a collision with it: G3 checks that +// an id is in range, not that something else reserved it, and G2 cannot see an id +// consumed inside a library call. Both report clean on colliding output. +// +// THE FAILURE MODE. Without a reservation the colourer can hand EVENT_ID0 to a +// compiler hazard whose interval SPANS the call. The library's own wait is then +// satisfied by that hazard's stale credit and its DMA drain silently does not happen +// -- correct-looking IR, wrong execution, no diagnostic anywhere. +// +// `seedHiddenMacroEvents` reserves those (direction, id) pairs over the macro's span, +// padded one SyncIR index each side, and the colourer treats a reservation as +// occupancy. `checkMacroHiddenEventCollisions` checks the result on every run. + +// THE FIX: the kernel compiles, and the collision is gone. +// RUN: ptoas --pto-arch=a3 --pto-level=level3 --enable-unified-sync %s 2>&1 >/dev/null | FileCheck %s --check-prefix=CLEAN --allow-empty +// CLEAN-NOT: event-id-collides-with-macro-hidden +// CLEAN-NOT: error: + +// THE ID MATCHES THE INCUMBENT'S, which is the property the reservation exists to +// restore. EVENT_ID1, not EVENT_ID0. If this ever prints EVENT_ID0 the reservation +// has stopped being consulted and the library's drain is silently unsynchronised. +// RUN: ptoas --pto-arch=a3 --pto-level=level3 --enable-unified-sync --emit-pto-ir %s | FileCheck %s --check-prefix=PINNED +// PINNED: pto.set_flag[, , ] +// PINNED: pto.wait_flag[, , ] + +// AND THE RESERVATION IS ACTUALLY SEEDED, not merely honoured by luck. A run that +// reserved nothing would also pass the two checks above if the colourer happened to +// pick EVENT_ID1 for its own reasons, so the count is pinned separately. +// RUN: ptoas --pto-arch=a3 --pto-level=level3 --enable-unified-sync --enable-unified-sync-debug %s 2>&1 >/dev/null | FileCheck %s --check-prefix=SEEDED +// SEEDED: [unified-sync coloring] interval coloring: {{.*}}hidden_reserved=7 + +// THE INCUMBENT is the reference answer and must be unchanged by any of this. +// RUN: ptoas --pto-arch=a3 --pto-level=level3 --enable-insert-sync --emit-pto-ir %s | FileCheck %s --check-prefix=INCUMBENT +// INCUMBENT: pto.set_flag[, , ] +// INCUMBENT: pto.wait_flag[, , ] + +module attributes {pto.target_arch = "a2a3"} { + func.func @mgather_row_macro_model(%arg0: !pto.ptr, %arg1: !pto.ptr, %arg2: !pto.ptr) + attributes {pto.kernel_kind = #pto.kernel_kind} { + %c0_i64 = arith.constant 0 : i64 + %c64_i64 = arith.constant 64 : i64 + %c64_index = arith.constant 64 : index + %c32_index = arith.constant 32 : index + %c1_index = arith.constant 1 : index + %c16_index = arith.constant 16 : index + %c0_index = arith.constant 0 : index + %mem_view = pto.make_tensor_view %arg0, shape = [%c64_index, %c32_index], strides = [%c32_index, %c1_index] {layout = #pto.layout}: !pto.tensor_view + %idx_view = pto.make_tensor_view %arg1, shape = [%c1_index, %c16_index], strides = [%c16_index, %c1_index] {layout = #pto.layout}: !pto.tensor_view + %out_view = pto.make_tensor_view %arg2, shape = [%c16_index, %c32_index], strides = [%c32_index, %c1_index] {layout = #pto.layout}: !pto.tensor_view + %idx_tile = pto.alloc_tile addr = %c0_i64 valid_row = %c1_index valid_col = %c16_index : !pto.tile_buf + %idx_pview = pto.partition_view %idx_view, offsets = [%c0_index, %c0_index], sizes = [%c1_index, %c16_index] : !pto.tensor_view -> !pto.partition_tensor_view<1x16xi32> + pto.tload ins(%idx_pview : !pto.partition_tensor_view<1x16xi32>) + outs(%idx_tile : !pto.tile_buf) + %dst = pto.alloc_tile addr = %c64_i64 valid_row = %c16_index valid_col = %c32_index : !pto.tile_buf + %mem_pview = pto.partition_view %mem_view, offsets = [%c0_index, %c0_index], sizes = [%c64_index, %c32_index] : !pto.tensor_view -> !pto.partition_tensor_view<64x32xf16> + pto.mgather ins(%mem_pview, %idx_tile + : !pto.partition_tensor_view<64x32xf16>, + !pto.tile_buf) + outs(%dst : !pto.tile_buf) + {coalesce = #pto} + %out_pview = pto.partition_view %out_view, offsets = [%c0_index, %c0_index], sizes = [%c16_index, %c32_index] : !pto.tensor_view -> !pto.partition_tensor_view<16x32xf16> + pto.tstore ins(%dst : !pto.tile_buf) + outs(%out_pview : !pto.partition_tensor_view<16x32xf16>) + return + } +} diff --git a/test/lit/pto/sync_unified_model.pto b/test/lit/pto/sync_unified_model.pto new file mode 100644 index 0000000000..a6e33b9249 --- /dev/null +++ b/test/lit/pto/sync_unified_model.pto @@ -0,0 +1,94 @@ +// Copyright (c) 2026 Huawei Technologies Co., Ltd. +// This program is free software, you can redistribute it and/or modify it under the terms and conditions of +// CANN Open Software License Agreement Version 2.0 (the "License"). +// Please refer to the License for details. You may not use this file except in compliance with the License. +// THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +// INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +// See LICENSE in the root of the software repository for the full text of the License. + +//===----------------------------------------------------------------------===// +// THE UNIFIED ALLOCATOR'S DATA MODEL. +// +// The premise of the allocator is that event ids, buffer ids and barriers are three +// RESOURCE CLASSES over one interval-colouring problem, not three algorithms. This +// file pins the problem statement: +// +// Hazard -- one ordering, OWNING BOTH HALVES of its sync pair +// Interval -- its live range in SyncIR-index space, the colourer's coordinates +// Resources -- an event pool per (src,dst); ONE shared buffer-id pool of K +// alpha -- the per-hazard mechanism cost, keyed on whether a reverse WAR exists +// +// Nothing routes to buffer-id on these kernels; what runs is per-direction first-fit +// interval colouring, reusing an id across disjoint intervals. +//===----------------------------------------------------------------------===// + +// --------------------------------------------------------------------------- +// 1. The model on a straight-line kernel: two pairs and the tail barrier. +// +// K=0 IS THE ARCHITECTURE, NOT A BRANCH. A3 is the same allocator with an empty +// buffer-id class; A5 opens it with K=32 in section 2. No arch test executes +// anywhere between the two runs -- K does all the work. +// RUN: ptoas --pto-arch=a3 --enable-unified-sync --enable-unified-sync-debug %S/Inputs/oracle_vadd.pto 2>&1 | FileCheck %s --check-prefix=VADD +// VADD: [unified-model] func=oracle_vadd hazards=3 K=0 + +// Intervals are SyncIR indices, not op positions -- the coordinate system +// MoveSyncState rewrites when it hoists. A barrier is a POINT (start == end), not a +// degenerate range. +// +// `clusters` is the buffer-id cluster the hazard's memory belongs to, joined on +// MemAlias. THE DISTINCTNESS IS THE POINT: h#0 (the load) and h#1 (the store) are on +// DIFFERENT buffers and must report DIFFERENT clusters. A join keyed on +// `(scope, baseAddr, size)` reports the same cluster for both, because +// `baseAddresses` is empty before addresses are assigned, so baseAddr collapses to 0 +// and distinct clusters share one key. If these two ever read the same cluster, the +// join has regressed to a degenerate key. +// `share` records whether the hazard's two endpoints are distinct allocations at one +// address, and whether those allocations have the same tile type (`reuse`) or +// differing ones (`alias`). It reads `none` here because this kernel assigns no +// explicit addresses -- they exist only under --pto-level=level3. The field is +// informational: no routing, refusal, mechanism choice or emission may read it. +// VADD-NEXT: h#0 pair MTE2->V iv=[1,2) pool=8 mech=event clusters=[1] revwar=no share=none alpha(e/b/bar)=1/2/100 event_id=0 +// VADD-NEXT: h#1 pair V->MTE3 iv=[2,3) pool=8 mech=event clusters=[0] revwar=no share=none alpha(e/b/bar)=1/2/100 event_id=0 +// VADD-NEXT: h#2 barrier ALL->ALL iv=[3,3) pool=8 mech=barrier clusters=[] revwar=no share=none alpha(e/b/bar)=1/2/100 event_id=- + +// Both pairs take id 0, and that is CORRECT rather than a collision: event flags are +// per-(src,dst) DISJOINT hardware, so id 0 in MTE2->V is a different flag from id 0 +// in V->MTE3. G2 keys on (src,dst,id) and does not fire. +// +// alpha prices buffer-id at 2 against event's 1 because revwar=no. A streaming kernel +// reuses no buffer in both directions, so a bidirectional token would buy nothing and +// would manufacture a reverse edge nobody asked for. Where revwar=yes, buffer-id +// drops to 1. +// VADD: [unified-sync coloring] interval coloring: assigned=2 overflow=0 spilled_to_barrier=0 hidden_reserved=0 reused=0 + +// The chosen ids are actually emitted, not merely computed. +// VADD: [unified-sync codegen] codegen WIRED: emitting set_flag/wait_flag/barrier from the colored model + +// --------------------------------------------------------------------------- +// 2. The same kernel with K=32: the SAME hazards and the SAME event assignment, with +// the buffer-id class now non-empty. +// +// The prefix is K32 rather than A5 deliberately. FileCheck scans the WHOLE file for +// `:`, so prose containing "A5:" anywhere in these comments would be read as +// a directive. +// RUN: ptoas --pto-arch=a5 --enable-unified-sync --enable-unified-sync-debug %S/Inputs/oracle_vadd.pto 2>&1 | FileCheck %s --check-prefix=K32 +// K32: [unified-model] func=oracle_vadd hazards=3 K=32 +// K32: [unified-sync coloring] interval coloring: assigned=2 overflow=0 spilled_to_barrier=0 hidden_reserved=0 reused=0 + +// --------------------------------------------------------------------------- +// 3. LOOP-CARRIED COMPENSATION, AND ID REUSE, on four tiles cycling in one loop. +// +// A loop-carried hazard's wait is satisfied by the previous iteration, so without a +// synthesised head-set the first iteration waits on a flag nothing raised and +// deadlocks. This is not an optimisation: `fe` counts the hazards the loop-carried +// predicate flagged, and `pairs` counts the head/tail groups appended. They must +// agree, one of each, and `linked` must match -- a synthesiser that primed once per +// loop rather than once per hazard would report fewer pairs than fe. +// +// `reused` being non-zero is the other half: four disjoint intervals per direction +// let the colourer hand the same id out more than once, which is what keeps a +// pressured kernel inside a pool of 8 instead of overflowing. +// RUN: ptoas --pto-arch=a3 --pto-level=level3 --enable-unified-sync --enable-unified-sync-debug %S/Inputs/unified_model_loop_carried.pto 2>&1 | FileCheck %s --check-prefix=LOOP +// LOOP: [unified-model] func=loop_carried_tiles hazards=17 K=0 +// LOOP: [unified-sync compensation] loop-compensation: fe=4 inv=4 pairs=4 heads=4 tails=4 linked=4 OK +// LOOP: [unified-sync coloring] interval coloring: assigned=12 overflow=0 spilled_to_barrier=0 hidden_reserved=0 reused=4 diff --git a/test/lit/pto/tassign_level3_loop_rebind.pto b/test/lit/pto/tassign_level3_loop_rebind.pto index 7c6f18b462..e964334846 100644 --- a/test/lit/pto/tassign_level3_loop_rebind.pto +++ b/test/lit/pto/tassign_level3_loop_rebind.pto @@ -1,5 +1,6 @@ // RUN: ptoas --pto-level=level3 --pto-arch=a5 %s | FileCheck %s // RUN: not ptoas --pto-level=level3 --pto-arch=a5 --enable-insert-sync %s 2>&1 | FileCheck %s --check-prefix=ERR +// RUN: not ptoas --pto-level=level3 --pto-arch=a5 --enable-unified-sync %s 2>&1 | FileCheck %s --check-prefix=UNIERR // RUN: not ptoas --pto-level=level2 --pto-arch=a5 %s 2>&1 | FileCheck %s --check-prefix=LV2ERR module { @@ -42,4 +43,9 @@ module { // CHECK: TPRINT{{(<.*>)?}}([[T]]); // ERR: pto.tassign requires --enable-insert-sync to be disabled +// Every automatic synchronization mode must refuse pto.tassign, not just the ones that +// happened to be listed first. The address a tassign rebinds is not modelled by the +// memory analysis all of these modes share, so a mode that accepts it silently emits +// synchronization computed against the wrong addresses. +// UNIERR: pto.tassign requires --enable-unified-sync to be disabled // LV2ERR: pto.tassign is only supported when --pto-level=level3. diff --git a/tools/ptoas/ptoas.cpp b/tools/ptoas/ptoas.cpp index 8fdaa31afa..6b843db251 100644 --- a/tools/ptoas/ptoas.cpp +++ b/tools/ptoas/ptoas.cpp @@ -397,6 +397,122 @@ static LogicalResult reorderEmitCFunctions(ModuleOp module) { // -------------------------------------------------------------------------- // Command Line Options // -------------------------------------------------------------------------- +static llvm::cl::opt dumpSyncExtract( + "dump-sync-extract", + llvm::cl::desc("Render the structured extraction of whatever sync the selected " + "mode emitted. Read-only"), + llvm::cl::init(false)); + +static llvm::cl::opt enableUnifiedSync( + "enable-unified-sync", + llvm::cl::desc("Enable the unified intra-core sync allocator: event ids, buffer " + "ids and barriers as three resource classes over one " + "interval-colouring problem"), + llvm::cl::init(false)); + +static llvm::cl::opt enableUnifiedSyncDebug( + "enable-unified-sync-debug", + llvm::cl::desc("Print the unified allocator's model, routing, colouring and " + "emission reports to stderr. Off by default; gate violations are " + "reported either way"), + llvm::cl::init(false)); + +static llvm::cl::opt unifiedSyncForceMechanism( + "unified-sync-force-mechanism", + llvm::cl::desc("Test-only: route hazard group 0 off its TYPE-derived mechanism " + "(bufid | bufid-spill), so the buffer-id path and the " + "resource-exhaustion downgrade can be exercised on a kernel the " + "router would otherwise leave alone"), + llvm::cl::init(""), llvm::cl::Hidden); + +static llvm::cl::opt autoBufidSingleCore( + "auto-bufid-single-core", + llvm::cl::desc("With --enable-insert-sync on a5: convert the WHOLE kernel to " + "buffer-id tokens when static dispatch is single-core, and " + "leave dual-subcore vector kernels on events untouched. " + "MEASURED: 12 single-core kernels, ALL 12 faster (+1.4% to " + "+55.2%; 3 of the 12 are one triplicated splitk body). " + "Dual-subcore is declined, and the evidence says that is " + "SOUND rather than merely cautious: collapsing duplicate " + "op-sequence bodies and excluding sub-microsecond deltas (no " + "A5 noise floor is established, so 0.01-0.43us moves are not " + "judgeable) leaves 1 WINNING shape against 4 LOSING shapes. " + "OFF by default"), + llvm::cl::init(false), llvm::cl::Hidden); + +static llvm::cl::opt checkAddrReuseWar( + "check-addr-reuse-war", + llvm::cl::desc("List the synchronization orderings whose two endpoints are " + "distinct tile allocations placed at the same physical " + "address in one scope. Reports; never fails"), + llvm::cl::init(false)); + +static llvm::cl::opt dumpSyncCoverage( + "dump-sync-coverage", + llvm::cl::desc("Write the happens-before edges this run's emitted sync induces, " + "as the reference profile for --check-sync-coverage"), + llvm::cl::init(false)); + +static llvm::cl::opt checkSyncCoverage( + "check-sync-coverage", + llvm::cl::desc("Run oracle gate G1: assert this run establishes every ordering " + "in the given file (a prior --dump-sync-coverage run). " + "DIFFERENTIAL -- its strength is bounded by that reference"), + llvm::cl::init(""), llvm::cl::value_desc("profile-file")); + +static llvm::cl::opt dumpSyncCounts( + "dump-sync-counts", + llvm::cl::desc("Write this run's kernel-body sync-op counts, one line per " + "function, as the reference for --check-sync-count-floor"), + llvm::cl::init(false)); + +static llvm::cl::opt checkSyncCountFloor( + "check-sync-count-floor", + llvm::cl::desc("Run oracle gate G4: assert this run does not synchronize more " + "than the given reference profile, per class (total, barriers, " + "PIPE_ALL barriers). DIFFERENTIAL"), + llvm::cl::init(""), llvm::cl::value_desc("profile-file")); + +static llvm::cl::opt checkSyncIds( + "check-sync-ids", + llvm::cl::desc("Run oracle gate G3: verify every static emitted sync id is " + "legal for its direction, op and arch. Rotating set_flag_dyn / " + "wait_flag_dyn ids resolve at runtime and are not checked " + "(fails on violation)"), + llvm::cl::init(false)); + +static llvm::cl::opt checkSyncIdsInjectFault( + "check-sync-ids-inject-fault", + llvm::cl::desc("Gate self-test: feed G3 a synthetic illegal id " + "(event-oor | block-all-stomp | bufid-oor). Appended to the " + "gate's own record list after extraction, never to the IR, " + "because an out-of-range event id is unrepresentable in IR"), + llvm::cl::init(""), llvm::cl::Hidden); + +static llvm::cl::opt checkSyncInterferenceInjectFault( + "check-sync-interference-inject-fault", + llvm::cl::desc("Gate self-test: judge a synthetic pre-codegen shape " + "(comp-conflict | multi-id-conflict) on its own record list. " + "Never added to the IR, so it cannot reach codegen"), + llvm::cl::init(""), llvm::cl::Hidden); + +static llvm::cl::opt checkSyncInterference( + "check-sync-interference", + llvm::cl::desc("Run oracle gate G2: assert no two overlapping intervals " + "share a sync id (fails on nesting / leak)"), + llvm::cl::init(false)); + +static llvm::cl::opt checkSyncSelfCoverage( + "check-sync-self-coverage", + llvm::cl::desc("Run oracle gate G1-self: assert every dependency the " + "front-end analysis (DepBetween) reports is ordered by the " + "emitted sync, except pairs on mutually exclusive scf.if arms, " + "PIPE_S->PIPE_S pairs on every arch, and, on A5, " + "PIPE_V->PIPE_V pairs the target orders itself; all three " + "exclusions are counted and reported. ABSOLUTE -- no " + "reference profile and nothing inherited from the sync pass"), + llvm::cl::init(false)); + static llvm::cl::opt enableInsertSync("enable-insert-sync", llvm::cl::desc("Enable automatic synchronization insertion pass"), llvm::cl::init(false)); @@ -789,6 +905,50 @@ static LogicalResult validateReserveBufferBase(pto::ReserveBufferOp op, return success(); } +/// Is this kernel dispatched to ONE core? +/// +/// A5 splits the vector core into two subcores, so a kernel_kind=Vector kernel +/// runs on BOTH (the trace shows core0.veccore0 AND core0.veccore1) while a Cube +/// kernel runs on one. This reads the same attribute PTOToEmitC reads to decide +/// whether to emit the __DAV_VEC__ guard. +/// +/// WHY IT GATES BUFFER-ID: whole-kernel token conversion always removes event +/// ISSUE overhead (measured: instruction-sum falls in 9 of 9 kernels), but a +/// token's block is welded to its access and cannot be repositioned the way +/// MoveSyncState repositions an event wait, and ids are buffer-keyed so +/// independent orderings coalesce onto one id. Where sync issue overhead +/// dominates the critical path that trade wins outright; where there is real +/// concurrency to lose it does not. +/// +/// EVIDENCE. Single-core is uniform: 12 measured, all 12 faster. Dual-subcore, +/// deduplicated by op sequence and restricted to deltas above the sub-microsecond +/// band, is 1 WINNING shape against 4 LOSING shapes -- so declining it is SOUND, +/// not merely conservative. +/// +/// Two corrections that produced that number, both worth keeping. The single +/// dual-subcore win (+46.9us) appears in FOUR distinct kernels -- different +/// sources, different emitted code -- that share one identical op sequence and +/// report identical spans to 0.01us; it is one result sampled four times, not +/// four results. And six further kernels move by only 0.01-0.43us, which cannot +/// be called either way because no A5 noise floor has been measured (the 0.000% +/// floor on record is A3-only). +/// +/// This gate does forgo that one winning shape. Capturing it needs a finer +/// predicate: whether the removed event overhead sat on the critical path is a +/// span property needing a run, and no static signal tested separates that shape +/// from the losers -- prelu-pto (+41.7%) and rem-pto (-1.6%) have identical +/// static profiles and land on opposite sides. +static bool isSingleCoreDispatch(ModuleOp module) { + bool anyDualSubcore = false; + module->walk([&](func::FuncOp fn) { + if (auto kk = fn->getAttrOfType( + pto::FunctionKernelKindAttr::name)) + if (kk.getKernelKind() == pto::FunctionKernelKind::Vector) + anyDualSubcore = true; + }); + return !anyDualSubcore; +} + static bool validateReserveBufferLevelRules(ModuleOp module, PTOBuildLevel level) { bool failed = false; @@ -3285,12 +3445,12 @@ int mlir::pto::compilePTOASModule( } int enabledAutoSyncModes = - (enableInsertSync ? 1 : 0) + (enableBufidSync ? 1 : 0) + + (enableInsertSync ? 1 : 0) + (enableUnifiedSync ? 1 : 0) + (enableBufidSync ? 1 : 0) + (enableInjectBarrierAllSync ? 1 : 0) + (enableGraphSyncSolver ? 1 : 0); if (enabledAutoSyncModes > 1) { llvm::errs() << "Error: --enable-insert-sync, --enable-bufid_sync, " - "--enable-inject-barrier-all-sync, and " - "--enable-graph-sync-solver are mutually exclusive.\n"; + "--enable-inject-barrier-all-sync, --enable-graph-sync-solver, " + "and --enable-unified-sync are mutually exclusive.\n"; return 1; } if (hasTAssign && enableInjectBarrierAllSync) { @@ -3308,6 +3468,11 @@ int mlir::pto::compilePTOASModule( "disabled.\n"; return 1; } + if (hasTAssign && enableUnifiedSync) { + llvm::errs() << "Error: pto.tassign requires --enable-unified-sync to be " + "disabled.\n"; + return 1; + } bool hasUserPlannedMultiAddrs = false; module->walk([&](pto::AllocMultiTileOp op) { @@ -3493,8 +3658,36 @@ int mlir::pto::compilePTOASModule( // `pto.multi_tile_get` operations and keeps their slot identity for alias // and event-id analysis. // solvers, while BufidSync is A5-only get_buf/rls_buf synchronization. - if (enableInsertSync) { - if (emitMlirIR) + if (enableUnifiedSync) + pm.addNestedPass( + pto::createPTOUnifiedSyncPass(arch == "a5" ? 32u : 0u, + enableUnifiedSyncDebug, + unifiedSyncForceMechanism, + checkAddrReuseWar)); + else if (enableInsertSync) { + // KERNEL-ENTIRE routing, not per-hazard: single-core dispatch takes + // whole-kernel buffer-id, dual-subcore keeps events (today's behaviour, + // byte-for-byte). Buffer-id is a5-only, so a3 always declines. + const bool singleCore = isSingleCoreDispatch(*module); + const bool autoBufid = autoBufidSingleCore && arch == "a5" && singleCore; + if (autoBufidSingleCore) + llvm::errs() << "[auto-bufid] dispatch=" + << (singleCore ? "single-core" : "dual-subcore") + << " arch=" << arch << " decision=" + << (autoBufid ? "BUFFER-ID" : "events (declined)") << "\n"; + if (autoBufid) { + // Mirror the --enable-bufid_sync branch exactly, --emit-pto-ir included: + // routing here must be indistinguishable from asking for buffer-id + // directly, on every output path and not just the emitted C++. + if (emitMlirIR) { + pm.addPass(std::make_unique( + SerialAutoSyncPass::Mode::Bufid, enableBufidSyncDebug, 0)); + } else { + PTOBufidSyncOptions options; + options.enableBufidSyncDebug = enableBufidSyncDebug; + pm.addNestedPass(pto::createPTOBufidSyncPass(options)); + } + } else if (emitMlirIR) pm.addPass(std::make_unique( SerialAutoSyncPass::Mode::InsertSync, false, 0)); else @@ -3531,6 +3724,31 @@ int mlir::pto::compilePTOASModule( // Materialize each `pto.multi_tile_get` as an addressed `pto.alloc_tile`; // dynamic selections use an `arith.select` chain over planned addresses. + // Oracle gates. Read-only apart from diagnostics, and each fails the pipeline on + // a violation rather than reporting and continuing. + if (checkSyncIds) + pm.addNestedPass(pto::createPTOCheckSyncIdsPass( + arch == "a5" ? 32u : 0u, checkSyncIdsInjectFault)); + if (checkSyncInterference) + pm.addNestedPass( + pto::createPTOCheckSyncInterferencePass( + checkSyncInterferenceInjectFault)); + if (checkSyncSelfCoverage) + pm.addNestedPass( + pto::createPTOCheckSyncSelfCoveragePass()); + if (dumpSyncExtract) + pm.addNestedPass(pto::createPTODumpSyncExtractPass()); + if (dumpSyncCoverage) + pm.addNestedPass(pto::createPTODumpSyncCoveragePass()); + if (!checkSyncCoverage.empty()) + pm.addNestedPass( + pto::createPTOCheckSyncCoveragePass(checkSyncCoverage)); + if (dumpSyncCounts) + pm.addNestedPass(pto::createPTODumpSyncCountsPass()); + if (!checkSyncCountFloor.empty()) + pm.addNestedPass( + pto::createPTOCheckSyncCountFloorPass(checkSyncCountFloor)); + pm.addPass(pto::createPTOResolveBufferSelectPass()); if (effectiveBackend == PTOBackend::EmitC) pm.addPass(createNarrowUnusedMultiResultProvenancePass());