Skip to content

feat: add Youtu-VL XLA vision and MLA inference (#872)#919

Draft
inureyes wants to merge 2 commits into
mainfrom
feature/issue-872-youtu-vl-xla
Draft

feat: add Youtu-VL XLA vision and MLA inference (#872)#919
inureyes wants to merge 2 commits into
mainfrom
feature/issue-872-youtu-vl-xla

Conversation

@inureyes

@inureyes inureyes commented Jul 24, 2026

Copy link
Copy Markdown
Member

Summary

  • Add an explicit Youtu-VL XLA text configuration that maps native dense DeepSeek-style MLA factors, compressed KV cache rows, and traditional interleaved RoPE across token/embedding prefill, single decode, and ragged continuous batching.
  • Add static 16/64/256-patch IREE vision buckets with checked grid planning, 2D RoPE, per-image/window attention masks, full-attention block selection, output-order restoration, and the checkpoint built-in merger without an extra projector.
  • Add canonical filtered loading and host preprocessing that enforces processor patch caps before allocation/upload, replaces configured image placeholders, preserves video-placeholder fallback only when image placeholders are absent, and rejects silent MLX fallback.
  • Bind processor identity, spatial/window/merger configuration, checkpoint weight schema, placeholder IDs, MLA decomposition, RoPE flags, and static limits into runtime compatibility boundaries.

Validation

  • cargo check --features xla-iree
  • cargo test -p mlxcel-xla --lib youtu_vl (focused XLA plan/emitter tests passed)
  • cargo test --features xla-iree --lib youtu_vl (21 passed)
  • cargo test --features xla-iree --lib youtu_vl_placeholder_selection (1 passed)
  • cargo test -p mlxcel-xla --lib youtu_vl_selects_dense_mla_and_interleaved_rope (1 passed)
  • Pinned Youtu-VL MLA token prefill, single decode, and ragged decode graphs compiled successfully with the pinned IREE compiler.
  • One pinned-checkpoint local-task IREE vision execution completed successfully with finite [4, 2560] merged output from the 16-patch bucket.
  • cargo clippy --features xla-iree --lib --no-deps
  • Central review follow-up e3383695 rejects unsupported attention_bias=true before MLA emission; the focused parser regression passed.
  • cargo test --lib reverse_indices_restore_original_group_order (1 passed with a non-self-inverse window permutation)
  • cargo fmt --all -- --check
  • git diff --check

Draft limitation

  • The checkpoint smoke test validates one real 16-patch local-task IREE execution, but a full deterministic token-level comparison against an independent reference oracle is not yet available; keep this PR in draft until that boundary is established or explicitly waived.

Test-suite note

  • The complete mlxcel-xla library suite has one pre-existing unrelated DeepStack C ABI assertion failure; 231 tests passed, 11 were ignored, and all focused Youtu-VL tests passed.

Closes #872

Add checked static patch buckets, window/full attention isolation, 2D vision RoPE, the built-in merger, and resident IREE checkpoint loading for Youtu-VL.

Map the dense DeepSeek-style MLA weights and compressed KV cache into token prefill, embedding prefill, single decode, and ragged continuous batching while retaining traditional interleaved RoPE.

Wire filtered host preprocessing, placeholder selection, sequence export, and strict no-MLX-fallback capability routing; fix window-order restoration and preserve pre-allocation patch caps.

Validated with xla-iree check, focused unit tests, clippy, pinned MLA IREE graph compilation, and one pinned checkpoint IREE vision execution.

Closes #872
@inureyes inureyes added status:review Under review type:enhancement New features, capabilities, or significant additions priority:low Low priority area:models Model architectures, weights, loading, metadata area:inference Generation, sampling, decoding (incl. speculative, DRY) labels Jul 24, 2026
Fail closed when a Youtu-VL MLA checkpoint requests attention projection biases because the emitter does not yet load or apply the q_a, kv_a, and output bias tensors.

Strengthen the vision restoration regression with a non-self-inverse window permutation so the prior incorrect permutation cannot satisfy the test accidentally.

Validation: cargo fmt --all -- --check; cargo test -p mlxcel-xla --lib youtu_vl_selects_dense_mla_and_interleaved_rope; cargo test --lib reverse_indices_restore_original_group_order.

Refs #872
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:inference Generation, sampling, decoding (incl. speculative, DRY) area:models Model architectures, weights, loading, metadata priority:low Low priority status:review Under review type:enhancement New features, capabilities, or significant additions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(xla): support Youtu-VL windowed vision and MLA inference

1 participant