feat: add Youtu-VL XLA vision and MLA inference (#872)#919
Draft
inureyes wants to merge 2 commits into
Draft
Conversation
Add checked static patch buckets, window/full attention isolation, 2D vision RoPE, the built-in merger, and resident IREE checkpoint loading for Youtu-VL. Map the dense DeepSeek-style MLA weights and compressed KV cache into token prefill, embedding prefill, single decode, and ragged continuous batching while retaining traditional interleaved RoPE. Wire filtered host preprocessing, placeholder selection, sequence export, and strict no-MLX-fallback capability routing; fix window-order restoration and preserve pre-allocation patch caps. Validated with xla-iree check, focused unit tests, clippy, pinned MLA IREE graph compilation, and one pinned checkpoint IREE vision execution. Closes #872
Fail closed when a Youtu-VL MLA checkpoint requests attention projection biases because the emitter does not yet load or apply the q_a, kv_a, and output bias tensors. Strengthen the vision restoration regression with a non-self-inverse window permutation so the prior incorrect permutation cannot satisfy the test accidentally. Validation: cargo fmt --all -- --check; cargo test -p mlxcel-xla --lib youtu_vl_selects_dense_mla_and_interleaved_rope; cargo test --lib reverse_indices_restore_original_group_order. Refs #872
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
cargo check --features xla-ireecargo test -p mlxcel-xla --lib youtu_vl(focused XLA plan/emitter tests passed)cargo test --features xla-iree --lib youtu_vl(21 passed)cargo test --features xla-iree --lib youtu_vl_placeholder_selection(1 passed)cargo test -p mlxcel-xla --lib youtu_vl_selects_dense_mla_and_interleaved_rope(1 passed)[4, 2560]merged output from the 16-patch bucket.cargo clippy --features xla-iree --lib --no-depse3383695rejects unsupportedattention_bias=truebefore MLA emission; the focused parser regression passed.cargo test --lib reverse_indices_restore_original_group_order(1 passed with a non-self-inverse window permutation)cargo fmt --all -- --checkgit diff --checkDraft limitation
Test-suite note
mlxcel-xlalibrary suite has one pre-existing unrelated DeepStack C ABI assertion failure; 231 tests passed, 11 were ignored, and all focused Youtu-VL tests passed.Closes #872