translator: expand multi-architecture coverage from ecosystem discovery

[cpunion]

Summary

• Expand Plan 9 assembly lowering across every supported architecture (386, amd64, arm, arm64, and wasm) using Go 1.27 tables, with positive/negative operand-form tests and LLVM 22 object checks.
• Make ecosystem discovery durable and incremental: every inspected exact version is recorded, assembly inventories are retained for future architectures, and failures remain retryable.
• Translate and LLVM-compile every applicable assembly file from all 838 discovered assembly-bearing exact versions in 32 deterministic shards.
• Keep the 28 user-issue/curated modules as a separate fast regression corpus.
• Use LLVM 22 only. Missing LLVM tools, unsupported instructions, parse errors, and compilation errors are hard failures.

Discovery funnel

The committed checkpoint is one manifest.json plus 256 sha256(module)[0] JSONL record shards. Records are deduplicated and sorted by module path, Go semantic version, and record kind. No run directories, gzip files, or ZIPs are committed.

Stage	Count	Meaning
Official index records consumed	237,098	13 contiguous recorded ranges
Successfully inspected exact module@version records	95,506	Includes versions with no assembly, so they are reusable
Unique module paths encountered	95,749	Paths seen across successful and retryable outcomes
Exact versions containing assembly	838	Generic discovery result, not a curated whitelist
Saved assembly paths	10,355	Used to query future architectures without rescanning the index
Retryable failure records	4,199	Never marked complete; retained for later retry
Discovery corpus candidates selected	838	Exactly once across 32 stable shards
Passed candidates	683	At least one applicable target translated and LLVM-compiled
Evidence-backed source N/A candidates	155	Go 1.27.1 rejects the source/package for every applicable supported target
Failed candidates	0	No instruction, parse, translation, LLVM, or infrastructure failure remains
Successful LLVM 22 translations	11,700	Per-file/per-target translations
Evidence-backed target N/A translations	50	Structured target/build/ABI rejection retained in reports

The invariant is 838 = 683 + 155 + 0; the aggregate verifier also proves every ledger match occurs once, in its correct shard, with the exact saved assembly inventory. The direct pass rate is 81.5036%; complete pass-or-evidence classification is 100%.

Public-index scope

A separate fixed-cutoff census counted 55,128,672 default-index module@version entries and 2,856,345 unique module paths.

Scope	Available at cutoff	Recorded here	Coverage
Default-index entries	55,128,672	237,098	0.430081%
Unique module paths	2,856,345	95,749	3.352151%

This checkpoint covers the contiguous index interval from 2026-09-13T14:59:13.837766Z through 2026-09-15T16:56:51.924668Z. Historical backfill to 2019 is not complete, so neither percentage is presented as whole-ecosystem completion.

Independently rediscovered issue libraries

These were selected by the generic ledger path, independently of the curated manifest:

Issue	Exact version	Saved assembly	Discovery outcome
llgo#2464	github.com/coder/websocket@v1.8.15	2 files	shard 7: passed, 6 translations
llgo#2552	github.com/klauspost/compress@v1.20.0	14 files	shard 28: passed, 42 translations
llgo#2576	github.com/tmthrgd/go-hex@v0.0.0-20190904060850-447a3041c3bc	2 files	shard 28: passed, 6 translations

The separate 28-library curated corpus also passes its complete 11-target matrix.

Official Go instruction coverage

Two different ratios are reported deliberately:

1. Encoder observation asks how much of Go's architecture encoder table is exercised by the official assembler corpus.
2. Lowerable forms asks how many observed non-context forms plan9asm currently lowers.

Arch	Encoder opcodes observed	Observation	Supported / context / unsupported forms	Lowerable non-context forms
386	20 / 1,603	1.25%	50 / 6 / 1	98.04%
amd64	1,451 / 1,603	90.52%	5,865 / 6 / 869	87.10%
arm	129 / 144	89.58%	382 / 42 / 75	83.59%
arm64	1,187 / 1,253	94.73%	854 / 52 / 1,074	44.29%
wasm	63 / 463	13.61%	0 / 120 / 0	context-only

Across the non-context rows, current lowerability is 7,151 / 9,170 = 77.98%. Relative to current upstream main, this PR adds 5,574 supported official form rows (1,577 → 7,151) and reduces unsupported rows by 5,593 (7,612 → 2,019). All five current reports have zero parse errors.

The low 386/wasm observation ratios demonstrate why the Go standard library and compiler tests alone cannot be treated as the full encoder specification; external discovery and direct encoder-table/form tests remain necessary.

The ARM64 x/arch decoder corpus has 1,108 forms: 716 supported, 63 context, 329 decoder-only/unsupported, and zero parse errors. For the required families that the Go 1.27 assembler itself accepts, the gate is complete: 184 forms = 146 supported + 38 context + 0 unsupported.

Major completed instruction families include x86 scalar/vector arithmetic, moves, shifts, comparisons, shuffle/permute/blend/broadcast, conversion/rounding, x87, AES/CLMUL, AMD SVM system-management (VMRUN/VMMCALL/VMLOAD/VMSAVE/STGI/CLGI/SKINIT/INVLPGA), strings/port I/O and carry arithmetic; ARM scalar integer/floating arithmetic, conversion, shifts/div/mod, comparisons and transfers; ARM64 scalar/vector floating, conditional, widening multiply, matrix multiply, test/logical, divide/remainder, FP16 multiply and ABI/tail-call families; and wasm ABI/global/metadata plus F32/F64 unary lowering.

No xfail was added. The refreshed baselines record only reviewed positive coverage movement: compared with the prior branch baseline, 386 resolved 2 unsupported forms, amd64 158, ARM64 22, and the ARM64 x/arch corpus resolved 33.

Discovery mechanism

• Historical scans proceed newest-to-oldest and prefer higher module-path major before semantic version; future incremental scans still admit a newly published higher major/version.
• history_before and incremental_since are derived from exact contiguous index ranges, so gaps and overlaps fail validation.
• Completed exact versions are reused before ZIP requests. New ZIPs are inspected by HEAD/range reads first; only assembly-bearing versions are materialized for corpus execution.
• One writer owns the ledger; internal workers parallelize network inspection and a deterministic merge performs deduplication and semantic sorting.
• Every applicable file proceeds through current-Go package build/applicability, go vet -asmdecl evidence where relevant, translation, and LLVM 22 compilation. N/A cannot hide an unsupported form.
• The 60-minute per-candidate deadline covers download, Go checks, and every applicable target; timeout remains a hard shard failure. It gives large generated corpora such as reflectx and llamawasm2go enough measured CI headroom without skipping work.
• A future platform can query saved asm_files and replay only eligible exact versions without rereading the module index or revisiting no-assembly records.

Validation

Validation on the current head:

• all 32 discovery shards: 838 selected / 683 passed / 155 source N/A / 0 failed; aggregate ledger/report consistency verified
• root go test ./... -count=1 on Go 1.27, plus both nested modules
• Go 1.20/1.22 compatibility coverage for the new generic-layout path; the Go 1.27-only DIT corpus is release-tagged rather than skipped
• root and nested builds, gofmt, and git diff --check
• official Go assembler coverage gate for all five architectures
• Go 1.27 standard-library target/feature matrix, zero unsupported forms and parse errors
• ARM64 x/arch Plan 9 corpus and native-Go-accepted required-family gate
• all 28 curated libraries across 11 targets
• native Go/LLVM semantic conformance for AMD64 and ARM64
• compile benchmark: 184/184 successful, 0 N/A; 18 seconds target total versus a 900-second budget
• Linux cross-execution is intentionally left to the qiniu linux/amd64 CI runner; the local darwin/arm64 driver fails explicitly instead of skipping

GitHub Actions is green (93/93 jobs), including 32/32 discovery shards and aggregate ledger verification. Codecov patch coverage is green at 87.62% against an 87.59% target. This PR remains Draft until review approval is complete.

[codecov]

Codecov Report

❌ Patch coverage is 90.51300% with 270 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
arm64_lower_atomic.go	89.06%	27 Missing ⚠️
amd64_raw_mul_low.go	82.69%	18 Missing ⚠️
arm64_lower_sve_absolute_difference.go	83.00%	17 Missing ⚠️
amd64_abi.go	76.19%	15 Missing ⚠️
amd64_lower_reciprocal.go	91.27%	15 Missing ⚠️
amd64_raw_same_width_conversion.go	89.05%	15 Missing ⚠️
amd64_lower_branch.go	66.66%	14 Missing ⚠️
arm64_lower_integer_pair.go	87.23%	12 Missing ⚠️
amd64_lower_fp.go	91.60%	11 Missing ⚠️
arm64_lower_sve_add_reduction.go	89.01%	10 Missing ⚠️
... and 32 more

📢 Thoughts on this report? Let us know!

[fennoai]

Review summary

Large PR (~175k insertions across 518 files). The bulk is generated discovery testdata (testdata/discovery/ledger/**) and highly repetitive AMD64/ARM instruction-lowering tables, both of which I skimmed rather than reviewed line-by-line. I focused on the high-signal changes: the new discovery/scan tooling (cmd/plan9asmcorpus/discovery.go, cmd/plan9asmdiscover, cmd/plan9asmll, cmd/plan9asmscan), internal/discoverymeta, and the core translate/parser changes.

Overall this is solid, defensively-written code: file handles are consistently closed on error paths, zip extraction blocks zip-slip (proxy-prefix + path.Clean + filepath.Rel + O_EXCL), external commands use fixed argv with escaped module paths, and remote fetches are size-bounded. Doc/AGENTS.md claims (flags, scripts, committed manifest counts, test names) were verified accurate.

One inline correctness/consistency finding below, plus a few non-blocking notes.

Non-blocking notes:

• Exponential build-tag search — cmd/plan9asmcorpus/discovery.go (findDiscoveryBuildTags). The search iterates 1 << len(customTags) masks, each doing filesystem-backed ctx.MatchFile calls, and the outer wantedCount loop re-walks the mask space ~len(customTags)+1 times. With the 16-tag cap this is bounded but can reach millions of MatchFile calls for a package near the limit. Usually fine (tags are typically 0–2); consider iterating masks once (prioritizing by popcount) if large-tag modules become common.

• Symlink escape in local include expansion — cmd/plan9asmll/main.go (pathWithinRoot / expandAsmIncludes). Containment is checked lexically via filepath.Rel; a #include target that is a symlink pointing outside the source root would pass the check and be read (no filepath.EvalSymlinks). Blast radius is contained (read-only, inlined into a discarded IR artifact, ephemeral CI against public modules; the zip extractor already rejects non-regular files), so this is defense-in-depth only. Resolving symlinks before the containment check would close the gap.

• Readability — cmd/plan9asmll/main.go:988 (and :1023). op == "JMP" || op == "B" || op == "RET" && len(ins.Args) == 1 relies on && binding tighter than ||. The current behavior is correct, but explicit parentheses around (op == "RET" && len(ins.Args) == 1) would guard against a future || term silently changing the grouping.