Apertus speculative decoding

This repository studies whether speculative decoding should be enabled for swiss-ai/Apertus-v1.5-70B on a four-GH200 Clariden node. It compares the plain 70B server, 8B draft-model speculation, and model-free n-gram speculation under controlled, repeated, production-facing workloads.

The next experiments are specified in the diagnostics and improvement plan and the EAGLE 3.1 execution plan: 8B first, then 70B. The EAGLE plan starts with a bounded Apertus 1.5 8B integration pilot, retaining EAGLE-3 as a fallback for a demonstrated 3.1-specific issue. Stage A (8B) is implemented: training pipeline in methods/eagle/apertus_eagle/, stage-aware launchers, and results in docs/eagle-8b-report.md. Progress is in docs/eagle-progress.md. A 3× decode speedup is a stretch target. Each target needs its own trained head; the public Apertus-8B-Instruct-2509 head is not a substitute for either Apertus 1.5 target.

The completed 8B drafter KV sparsity study reduced physical drafter storage but failed the throughput and acceptance gates. Its opt-in serving overlay is in serving/draft_kv/, its campaign tools are in experiments/drafter-kv-8b/, and raw measurements and provenance are under results/8b/drafter-kv/.

The 8B wasted-compute profiling study measured with Nsight Systems how much GPU time speculation wastes past saturation: at k=7 and C=128 about 22% of GPU time and 45-46% of the work speculation adds. A load-dependent k is worth about 4% geomean over the best fixed k, and the DSpark paper scheduler adds only 1.9-2.8% over plain at C=128-256 on summarization, too little to justify another serving mechanism. The opt-in NVTX overlay is serving/patches/vllm-apertus-nvtx.patch, the harness support is in src/apertus_bench/{profiling,vllm_nvtx,waste_report}.py, raw sweeps are under results/8b/sweeps/ and analyses under results/8b/waste/.

The 8B multi-drafter tree screen (TreeGraft-style trees from DSpark, EAGLE and n-gram candidates, replayed offline against greedy target captures) found low-load headroom from a single DSpark tree but no case for multi-drafter serving: grafting n-gram candidates changes projected throughput by less than 0.1%, running both neural drafters is 10-27% slower, and no tree gain remains at C>=64. The simulator is src/apertus_bench/tree_sim.py, the candidate builders are in methods/tree/, and the evidence is under results/8b/tree-drafters/.

The 8B serving-layout campaign measured the final DSpark and EAGLE heads under DP4, TP4, TP2DP2 and prefill/decode disaggregation, thinking off and on, up to C=1024. Speculation is a load-specific choice with material memory costs: TP4 k7 gives 1.44-2.64x paired throughput at C1024 but regresses Arena-Hard 28-45% at C128 and uses about 70 GiB more resident memory per node; TP2DP2 k3 gains 44-50% at C128. P/D is not worth implementing for this decode-heavy grid. Specs are in experiments/high-concurrency-8b/, raw sweeps under results/8b/sweeps/, and the recovered runs and reviews under results/8b/layout-recovery-*/.

The final 8B sampling sweeps found no case for a narrower sampler top-k as a speed optimization (top-k5 changes throughput by -5% to +8%, top-k20/50 by at most +0.6%), while draft length should depend on load: short drafts help at intermediate load and k7 loses badly at high load. The validated analysis is in results/8b/sampling/recovery-20261027/.

The 8B online drafter adaptation study fine-tuned the EAGLE e10 head on served-workload traces and swapped heads into a running server without a restart. Adaptation is worth keeping in mind for persistent multilingual traffic (+17% throughput at k3/C1) and low-load summarization (+14%), but chat gains are insufficient and high-load gains fail the benefit rule, so no blanket replacement of the default drafter is justified. The swap and confirmation harness is in src/apertus_bench/{drafter_swap,drafter_reload,adaptation_confirmation}.py, training tools in methods/eagle/apertus_eagle/, configs in methods/eagle/configs/8b/adapt/, and evidence under results/8b/drafter-updates/.

The 8B shallow DSpark drafter study asked whether a 1-3 layer DSpark drafter wins under load by costing less per step. Each removed layer saves 1.35-2.2 ms of draft time at 256 requests, but the trained 1-layer drafter loses acceptance (held-out EAL 3.52 versus 4.63) and twice failed the calibrated greedy correctness gate in serving, so the study closed as correctness blocked with no speed or implementation claim. The depth serving overlay is serving/patches/vllm-apertus-dspark-num-layers.patch, the truncation, training campaign and cost models are in methods/dspark/, and evidence is under results/8b/small-drafter/ and results/8b/sweeps/small-trunc-*.

The project is scoped as an 8 ECTS semester research project. It goes beyond a one-off speed test: it measures latency, throughput, acceptance, serving capacity, memory trade-offs, and the conditions under which a configuration wins or loses.

Research questions

1. How does speedup vary across open chat, code, and long-context summarization at concurrency 1, 8, and 32?
2. How strongly do draft acceptance rate and mean accepted length predict TTFT, TPOT, and output throughput?
3. Which combination of speculative depth {2,3,5,8} and draft TP is best, and when does draft overhead outweigh accepted work? Draft TP is pinned to 4 for now: the pinned vLLM refuses draft TP != target TP, so only a depth sweep is answerable (results/70b/deployment-failures/draft-n3-tp1-3392110/).
4. When is n-gram speculation a better operational choice than the 8B draft model?
5. Does the winning setting increase sustainable request rate without unacceptable tail latency or loss of KV-cache capacity?

The full experimental contract is in docs/protocol.md, the initial reading map is in docs/literature.md, and the staged matrix is machine-readable in experiments/smoke-70b/experiment.yaml.

Current findings: docs/hackathon-20260915.md is the standing summary of what has been measured — the headline result, the step-cost decomposition that explains it, the platform bugs it surfaced, the diff against the published model-launch speculative-decoding example, and what remains unanswered.

What is implemented

• parameterized baseline, draft-model, and n-gram sml launchers using Capstor model paths;
• a streaming OpenAI-compatible client measuring per-request E2E, TTFT, conventional TPOT, and stream-event gaps;
• before/after snapshots of vLLM's exact speculative-decoding Prometheus counters;
• closed-loop fixed-concurrency matrix execution with warmup and deployment-level repeats;
• sequential greedy-output capture, so correctness can be checked while using only one model node, and a greedy correctness gate calibrated against same-configuration deployments;
• immutable raw cell artifacts plus a CSV analysis step;
• a small smoke corpus and a specified schema for the real benchmark corpus.

The current model-launch k6 path is useful for serving-capacity sweeps, but it sends non-streaming requests and therefore cannot produce true client-observed TTFT. This repository uses its own streaming client for latency experiments and reuses swiss-ai/bench-agent for the final open-loop capacity experiment.

Repository layout

Methods share one harness, one serving layer and one analysis step, so a speedup is always taken against a plain baseline launched the same way. Each method names its target explicitly.

src/apertus_bench/   benchmark client, correctness gate, analysis, EAGLE head validation
serving/             plain-target launcher, environment resolution, vLLM overlay patches
└── draft_kv/        opt-in DSpark sparse-cache runtime, installer and GPU checks
methods/
├── draft_model/     8B draft, 70B target
├── ngram/           prompt lookup
├── tree/            offline tree / multi-drafter candidate builders (analysis only)
└── eagle/           EAGLE 3.1 / EAGLE-3: trainer package, configs/<stage>/, launchers, TorchSpec patches
targets/<stage>/     target contracts (architecture, aux layers, hashes) and environment locks
experiments/         one-off campaigns: smoke-70b, round-cost-70b, system-8b, drafter-kv-8b,
                     high-concurrency-8b
workloads/           corpora; workloads/<stage>/ holds manifests (restricted text stays on the cluster)
results/<stage>/     raw cells and reports, by target (8b, 70b) and then method

Setup

Keep this repository next to a model-launch checkout at the pinned revision (or a reviewed newer revision containing the Apertus 1.5 release environment):

Projects/
├── apertus-speculative-decoding/
└── model-launch/

Create the local environment:

cd apertus-speculative-decoding
python -m venv .venv
. .venv/bin/activate
python -m pip install -e '.[dev]'
apertus-bench validate-workloads workloads/smoke.jsonl

If another branch currently occupies the main model-launch checkout, a separate Git worktree avoids disturbing it:

git -C ../model-launch fetch origin
git -C ../model-launch worktree add ../model-launch-apertus \
  909026a990454557f1b54d26f24ec3ad92e51e35
export MODEL_LAUNCH_ROOT=../model-launch-apertus

The launchers default to the cached Capstor paths. Override TARGET_MODEL, DRAFT_MODEL, or MODEL_LAUNCH_ROOT when the site layout differs.

Launch a variant

From this repository on the Clariden login node:

# Plain target
./serving/baseline.sh

# 8B draft, depth 5, draft TP=1
NUM_SPECULATIVE_TOKENS=5 DRAFT_TP=1 ./methods/draft_model/draft-model.sh

# N-gram, depth 5
NUM_SPECULATIVE_TOKENS=5 PROMPT_LOOKUP_MAX=4 ./methods/ngram/ngram.sh

# EAGLE-3 / EAGLE 3.1 (engine method is always eagle3; ALGORITHM selects the trained head).
# STAGE=8b: target TP=1, draft TP=1; STAGE=70b: 4/4. The head must match the stage's target
# contract; the public Apertus-8B-Instruct-2509 head is rejected.
STAGE=8b EAGLE_HEAD=/path/to/apertus15-8b-eagle31-head ALGORITHM=eagle31 NUM_SPECULATIVE_TOKENS=2 \
  ./methods/eagle/launch/eagle.sh --no-tui

Each launcher prints its unique served model name. Do not begin measurement merely because the name appears in /v1/models; require one successful short chat completion first.

The launchers speak the pinned model-launch CLI (--system, --framework, --environment, --nodes-per-replica, --time). Builds from before model-launch commit 4413441 call those --firecrest-system, --serving-framework, --slurm-environment, and that same commit renamed the OpenTela share mount from /ocfbin to /opentelabin, so an sml from the other side of it paired with the pinned environment toml fails on the node rather than at submission. Install the pin and put it first on PATH:

python3 -m venv ~/venvs/sml-apertus
~/venvs/sml-apertus/bin/pip install -e "$MODEL_LAUNCH_ROOT"
export PATH="$HOME/venvs/sml-apertus/bin:$PATH"

The packaged environment toml also carries a literal {arch} in its image path, which only some sml builds substitute on the node; one that does not makes pyxis reject the placeholder and the job dies seconds after it starts. serving/resolve-env.sh writes a resolved copy (arm64 for GH200) under ~/.sml and the launchers pass that. Override the source toml with ENV_SOURCE, the architecture with SML_ARCH, and the partition with SML_PARTITION; any extra arguments are forwarded to sml advanced, so --no-tui works for non-interactive launches.

draft_model speculation additionally needs a source fix that the pinned image predates: vLLM reads image_token_index off the target config, which Apertus 1.5 does not define (it uses image_token_id), so the drafter fails to load and the engine never starts. serving/patches/ holds the one-line fix and serving/patch-vllm.sh binds a corrected copy over the read-only image:

IMAGE=/capstor/store/cscs/swissai/infra01/container-images/ci/vllm_apertus_1.5_release-arm64.sqsh
EXTRA_MOUNTS="$(./serving/patch-vllm.sh "$IMAGE" serving/patches/vllm-apertus-image-token.patch)" \
  ./methods/draft_model/draft-model.sh --no-tui

Record the overlay alongside any measurement taken with it, and drop EXTRA_MOUNTS once a rebuilt image carries the fix. The baseline and n-gram variants never load a drafter, so they do not need it.

Greedy correctness gate on one node

Capture the baseline outputs, stop the baseline job, launch the speculative variant, and capture again with the same corpus and sampling settings:

export CSCS_SERVING_API='...'
API=https://api.swissai.svc.cscs.ch

apertus-bench capture \
  --base-url "$API" \
  --model "$BASELINE_SERVED_MODEL" \
  --workloads workloads/smoke.jsonl \
  --output results/70b/correctness/baseline.json

apertus-bench capture \
  --base-url "$API" \
  --model "$SPEC_SERVED_MODEL" \
  --workloads workloads/smoke.jsonl \
  --output results/70b/correctness/draft-n3-tp4.json

apertus-bench compare-captures \
  results/70b/correctness/baseline.json \
  results/70b/correctness/draft-n3-tp4.json \
  --output results/70b/correctness/comparison.json

The comparison alone decides nothing. Independently launched deployments do not reproduce each other's greedy tokens — two identically configured draft deployments agreed on 0 of 6 smoke prompts — so a mismatch is not evidence that speculation is lossy. Capture a calibration control as well: a second capture from the same deployment, and a capture from another deployment of one configuration. Then judge the comparison against it:

apertus-bench correctness-gate \
  --treatment results/70b/correctness/baseline.json results/70b/correctness/draft-n3-tp4.json \
  --control results/70b/correctness/draft-n3-tp4.json results/70b/correctness/draft-n3-tp4-repeat2.json \
  --output results/70b/correctness/gate.json

The gate passes when the two arms diverge no more than two deployments of one configuration do, which is consistent with losslessness rather than a proof of it, and it refuses to run without a control. docs/protocol.md states the criterion, both control kinds, and why a same-deployment paired check is impossible here.

Run one deployed variant

The example below runs all three smoke strata at concurrency 1 and 8. The metrics URL must resolve to that vLLM server's /metrics endpoint; use --no-metrics only for harness debugging.

apertus-bench matrix \
  --base-url "$API" \
  --metrics-url "$VLLM_METRICS_URL" \
  --model "$SPEC_SERVED_MODEL" \
  --variant draft-n3-tp4 \
  --method draft_model \
  --num-speculative-tokens 3 \
  --draft-tensor-parallel-size 4 \
  --workloads workloads/smoke.jsonl \
  --concurrencies 1 8 \
  --requests 32 \
  --repeats 2 \
  --metadata slurm_job_id="$SLURM_JOB_ID" \
  --output results/smoke/draft-n3-tp4

For the real experiment, replace the smoke corpus, use the matrix and repeat counts in the protocol, and put each independently launched deployment in its own result directory.

After baseline and candidate runs exist:

apertus-bench analyze results/experiment-001 --output analysis/results.csv

Metric definitions

• TTFT: client request start to the first non-empty streamed content/reasoning/tool payload.
• TPOT: (E2E - TTFT) / (completion_tokens - 1) per request; this is the conventional average inter-token latency measure used for the headline result.
• Stream-event gap: time between non-empty SSE payloads. Speculation can deliver several accepted tokens in one event, so this diagnostic is not treated as token-level ITL.
• Output throughput: successful completion tokens divided by whole-cell wall time.
• Acceptance rate: delta accepted draft tokens / delta proposed draft tokens.
• Mean acceptance length: 1 + accepted_tokens / drafts, including the verified bonus token, matching pinned vLLM semantics.

Requests within one deployment are useful for latency distributions but are not independent experimental replicates. Confidence intervals and configuration comparisons use independently launched deployment repeats.

Upstream pins

• swiss-ai/model-launch: 909026a990454557f1b54d26f24ec3ad92e51e35
• swiss-ai/vllm branch apertus-1-5: a601a9d998ddeb488f0c17e8512874b116aa7658
• vLLM base: v0.23.1rc1