feat: add Gemma3 VLM OpenXLA path (#869)

[inureyes]

Summary

• Add the pinned Gemma3 SigLIP tower and average-pool projector to the resident StableHLO/IREE vision module, including exact processor, projector, weight-shape, and artifact-identity contracts.
• Apply the SigLIP post_layernorm after the final encoder block on the IREE path, matching the eager MLX tower, and version the vision artifact identity for the corrected graph.
• Add the Gemma3 prepared-prefill seam with one-time text embedding scaling, native projector rows, and the authoritative bidirectional padding mask used by embeddings prefill.
• Bind the prepared-prefill ABI, additive-mask mode/value, post-scale policy, newline/image/pad/BOI/EOI tokens, SigLIP/projector configuration, and Gemma sliding/global settings into an explicit stable identity used by both single-sequence and ragged IREE bundle fingerprints. Preserve the existing dense identity byte-for-byte for ordinary text-only Gemma3 configs.
• Parse the nested Gemma3 text configuration for the shared language emitter and preserve local/global RoPE behavior without changing ordinary text-only token prefill.
• Treat root text-only Gemma3 configurations as direct language-model configs unless they also declare a vision configuration, preserving existing text-session startup while retaining nested VLM defaults.
• Route CLI and continuous-batch image requests through a filtered host embedding table and resident IREE projector, with no duplicate decoder and no silent host/LLaVA fallback. Preserve the merged Qwen2-VL route and reject unqualified wrapper configs conservatively.
• Preserve Gemma3Processor's exact newline, BOI, image-token, EOI, newline block for both existing placeholders and bare CLI/server prompts, with checked capacity accounting for every wrapper token.
• Add a pinned mixed-runtime boundary diagnostic that captures processor pixels, twelve block-0 sub-stages, selected/final SigLIP states, post-layernorm, average-pool projection, prepared image rows, one-time sqrt(hidden) scaling, and the exact bidirectional padding mask.
• Expose that diagnostic as the standalone xla_gemma3_reference_check CUDA example so it avoids the libtest harness, while retaining the ignored test as a thin wrapper and emitting flushed stage progress plus 60-second heartbeats.
• Configure IREE's diagnostics-only process-global task threading exactly once before the standalone runner creates its first local-task instance: one topology group and host-default pthread stacks, with cached success/failure and no shared production xla_aux/xla_iree startup change.
• Reuse the existing eager and IREE hidden-state captures at exact ordered block 6, 13, 20, and final-block descriptors, partitioning blocks 1 through 26 into four balanced first-failure intervals without changing the graph, tolerances, or production path.
• Refine the known block-1-through-6 interval by comparing block outputs 1 through 5 before the existing block-6 checkpoint and emitting the exact first failing block while preserving the later bisection report.

Validation

• cargo test -p mlxcel-xla --lib gemma3_ (8 passed)
• Gemma3 diagnostics comparator test (1 passed)
• cargo test -p mlxcel-xla prepared_gemma3 --lib (7 passed, including stable contract field sensitivity and unchanged text-only identity)
• cargo test -p mlxcel-xla --features iree gemma3_vlm_compatibility --lib (2 passed: wrapper contract binding and direct-root text-only identity)
• cargo test -p mlxcel-xla gemma3_avg_pool_projector_binds_pool_and_prompt_contract --lib (1 passed)
• cargo test -p mlxcel-xla qwen2_vl --lib after rebasing feat(xla): support Qwen2-VL as the representative M-RoPE VLM #865 (4 passed)
• cargo test --features xla-iree --lib host_preprocessor after rebasing feat(xla): support Qwen2-VL as the representative M-RoPE VLM #865 (13 passed)
• cargo test --lib vlm_prompt (16 passed, including bare Gemma3 wrapping and unchanged unwrapped LLaVA insertion)
• cargo test -p mlxcel-xla gemma3 --lib (52 passed, 2 ignored, including direct-root and nested-wrapper configuration regressions)
• unqualified llama, qwen2_vl, and gemma3 capability regression gate
• cargo check --features xla-iree --lib --bins
• cargo fmt --all -- --check
• IREE_DIST=... cargo check --no-default-features --features xla-reference-diagnostics --lib
• IREE_DIST=... cargo clippy --no-default-features --features xla-reference-diagnostics --lib (existing warnings only)
• Cargo metadata confirms xla_gemma3_reference_check requires both cuda and xla-reference-diagnostics.
• targeted Clippy for the corrected Gemma3 graph, diagnostics, and runtime compatibility contract (existing warnings only)
• Real CUDA IREE projector execution using /home/inureyes/models/gemma-3-4b-it-4bit: compiled and returned finite [256, 2560] projected features in 32.17 seconds.
• Post-fix production CUDA/IREE mlxcel generate run with the pinned checkpoint and image fixture completed successfully (k, 1 token, exit 0; reported generation time 8.11s).
• Independent deterministic HF eager versus IREE local-task Gemma3 VLM embeddings/mask oracle: logits max absolute difference 0.006675, greedy token exact (14), all-layer K/V within max 0.25, RMS 0.1, and cosine 0.99 envelopes; causal-mask, multiplicative-mask, double-scale, and missing-image-pre-scale negatives all diverged.
• Focused diagnostics-only threading regressions cover configuration-before-creation, error short-circuiting, exact two-flag source contract, native repeated-call reuse, and production xla_aux/xla_iree isolation (5 passed across filtered runs).
• The linked diagnostics libtest contains both xla_diagnostics_configure_local_task_threads and iree_flags_parse; the native reuse test proved the pinned 3.11 parser accepts and consumes both synthetic flags.
• IREE_DIST=... cargo check --no-default-features --features cuda,xla-reference-diagnostics --example xla_gemma3_reference_check
• IREE_DIST=... cargo test --features xla-reference-diagnostics --lib reference_boundary:: (10 passed, 1 actual ignored; includes exact balanced descriptors, ordered block-1-through-6 refinement descriptors, interval bounds, and exact-failure selection)
• The bounded standalone actual at c44147f8 kept processor pixels, siglip.hidden.embedding, and every block-0 sub-stage passing. siglip.hidden.block6.output failed first with 333 failures, max absolute difference 1.760086, and first failure at flat index 42197; block 13 failed with 35,808 failures and max absolute difference 1.969528, block 20 failed with 489,930 failures and max absolute difference 68.06642, and siglip.hidden.last_pre_layernorm failed with 1,408,110 failures and max absolute difference 1302.648. The coarse report resolved first_failing_block_range=1..=6.
• The bounded standalone actual at 002bb77e kept the prepared-prefill self/invariant and mask checks passing exactly. siglip.hidden.block1.output failed first with 8 failing values, max absolute difference 0.6672516, and first failure at flat index 658517; block 2 failed with 18 values and max absolute difference 0.7214432, block 3 with 61 and 1.209267, block 4 with 73 and 1.210052, block 5 with 133 and 1.209457, and block 6 with 333 and 1.760086.
• The ordered report resolved exact_first_failing_block=1. Exit 101 is the diagnostic's intended first-divergence panic at siglip.hidden.block1.output flat index 658517, not a worker-creation regression; numeric acceptance remains open. No further actual was run, and graph outputs, production behavior, tolerances, CSV handling, and diagnostic stack flags are unchanged.
• Padding/multi-image continuous batching, cancellation, and slot reuse remain required before this draft is ready.

Runtime notes

• Gemma3 image execution requires the xla-iree feature and rejects MLXCEL_XLA_VISION_BACKEND=host explicitly.
• The standalone mixed-runtime runner requires MLX CUDA at build time and pins only the IREE side to local-task; immediately before its first IREE instance it applies --task_topology_group_count=1 and --task_worker_stack_size=0 once through the diagnostics-only flag parser.
• Text-only Gemma3 stays on the ordinary token-prefill path and retains its pre-existing dense artifact identity.
• Runtime artifacts are cached outside the repository; no generated CSV or model artifact is committed.

Closes #869

[inureyes]

Rebased onto main@7fe1412d and force-pushed as c4e5d22d.

Post-rebase validation:

• cargo fmt --all -- --check
• git diff --check
• cargo test -p mlxcel-xla --lib gemma3_: 8 passed
• cargo check -p mlxcel-xla --features iree: passed with existing warnings

The shared vision runtime inherited #934’s explicit new_legacy_unqualified artifact path during the rebase; no additional branch-local adaptation commit was needed. The strict SigLIP block-1 numeric failure (8 values, prior max_abs=0.6672516) and continuous-batching gates remain unresolved. With CUDA unavailable on the current host, no fresh actual-checkpoint result is claimed. This PR remains draft and #869 remains blocked by #932.

[inureyes]

Rebase and bounded SigLIP evidence — 2026-07-27

• Rebased the feature branch onto origin/main@ec6fbb6d and force-with-lease pushed 9ab13992. The vision conflict adopts the shared numeric_ops LayerNorm/GELU/softmax helpers from main, the feature conflict preserves the CPU-capable xla-micro-oracle, and the diagnostics conflict adopts the shared one-group plus host-default-stack helper from main.
• The only authorized actual command was env IREE_CUDA_HOME=/home/inureyes/.cache/mlxcel/iree-cuda-3.12.0rc20260721 IREE_CUDA_COMPILE=/home/inureyes/.cache/mlxcel/iree-cuda-3.12.0rc20260721/venv/bin/iree-compile MLXCEL_GEMMA3_REVISION=93724907d4ed1745d2fe50baadf3b0b01a65abf2 RUSTC=/home/inureyes/.rustup/toolchains/1.93.1-aarch64-unknown-linux-gnu/bin/rustc /home/inureyes/.rustup/toolchains/1.93.1-aarch64-unknown-linux-gnu/bin/cargo run --locked --example xla_gemma3_reference_check --features cuda,xla-reference-diagnostics -- --model /home/inureyes/models/gemma-3-4b-it-4bit --image tests/fixtures/test_image.png --device local-task.
• That current-head actual was stopped at the bounded eight-minute cap before the model runner began. The new Rust 1.93.1 target caused mlxcel-core to classify its MLX cache as stale, purge _deps, and attempt the pinned MLX clone; the sandboxed attempt could not resolve GitHub. After allowing the required fetch, rebuilding pinned MLX revision b7c3dd6d27f45b5365b08a840310187dc503f1db completed and the same current-head example passed cargo check in 7m07s. No second actual or full-model run was started.
• Therefore there is no new current-head numeric result. The last completed actual remains the documented 002bb77e run: first divergence siglip.hidden.block1.output, flat index 658517, 8 failing values, max absolute difference 0.6672516; exact first failing block 1. This remains a numeric acceptance blocker and was not waived.
• Focused validation after the rebase: cargo check --locked --features cuda,xla-reference-diagnostics --example xla_gemma3_reference_check passed with existing warnings; cargo test --locked -p mlxcel-xla --lib gemma3 passed 54 with 2 ignored; cargo test --locked -p mlxcel-xla --lib diagnostic_flags::tests passed 2; cargo fmt --all -- --check passed.
• No tolerance, production graph, CSV, or additional numeric fix was introduced. The PR remains draft; SigLIP intermediate acceptance, final token-exact output, padding/multi-image continuous batching, cancellation, and slot reuse remain open.