Skip to content

benchmarks: move adaptive prefill calibration to Examples - #1018

Open
SuperGoodGame wants to merge 5 commits into
deepspeedai:masterfrom
SuperGoodGame:benchmarks/adaptive-prefill-calibration
Open

SuperGoodGame wants to merge 5 commits into
deepspeedai:masterfrom
SuperGoodGame:benchmarks/adaptive-prefill-calibration

Conversation

@SuperGoodGame

@SuperGoodGame SuperGoodGame commented Oct 10, 2026 •

Copy link
Copy Markdown

Adds adaptive prefill calibration and evaluation tooling alongside the existing rollout benchmark in benchmarks/opsd. Requires Transformers >=4.56 and DeepSpeed PR #8786; the README explains configuration loading, the short-context calibration range and deployment validation.

Calibration derives ordinary MHA KV heads from query heads when the dedicated field is absent. The CLI requires a complete metadata object for supplied calibration files; cost-only artifacts are rejected, while applications can still pass hand-tuned coefficients directly to the rollout configuration. Saved configurations are checked against the model/device/library metadata and a combined source hash of rollout, bucket planning and static-cache implementations; stale or missing hashes require recalibration.

Validation: three regression tests pass, covering non-negative fitting, real tiny-OPT calibration on a GPU, and CLI acceptance/rejection of matching, changed, missing and cache-only-changed source hashes. Six targeted GPU oracle checks in the DeepSpeed companion validate OPT/Qwen generation through the static cache. Python lint and diff checks pass. The original seeded tiny-Qwen2 calibration/sweep smoke check also completed all four workloads with matching output IDs on every path.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-11-05T05:42:09.433651Z
We couldn’t complete this review. Comment @codex review to try again.
8c73cbd New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dcdde0bf7f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread benchmarks/opsd/rollout_prefill.py Resolved
Comment thread benchmarks/opsd/rollout_prefill.py Resolved

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 79b2140eaf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread benchmarks/opsd/rollout_prefill.py Resolved
Comment thread benchmarks/opsd/rollout_prefill.py Resolved

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: beb3401dce

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread benchmarks/opsd/rollout_prefill.py Resolved

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3fd685fc55

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

'allow_tf32', 'source_sha256'):
if calibration['metadata'].get(key) != metadata[key]:
parser.error(f'calibration differs in {key}; recalibrate for this configuration')
costs = calibration.get('costs', {})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject incomplete saved cost maps

When --cost-config points to an otherwise matching calibration artifact whose costs member is absent or has lost one of the five emitted coefficients, this fallback silently supplies HybridEngineRolloutConfig defaults for the missing values and the sweep completes with current metadata as though the calibrated policy was evaluated. Since --calibrate always emits the complete map (and hand-tuned partial configurations are documented for direct application use), validate the required cost keys before running the sweep.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

A supplied calibration file must provide all five cost coefficients. A
missing cost map, or any missing coefficient, now fails the CLI instead
of silently falling back to defaults for that term.

The CLI regression checks a missing cost map and each missing key.

Signed-off-by: SuperGoodGame <[email protected]>
@SuperGoodGame
SuperGoodGame force-pushed the benchmarks/adaptive-prefill-calibration branch from ba438e5 to 8c73cbd Compare November 5, 2026 05:42
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@SuperGoodGame

Copy link
Copy Markdown
Author

Added 8c73cbd, which closes the remaining gap in the saved-calibration check: a supplied cost map must now carry all five coefficients. A missing costs object, or any missing key, fails the CLI instead of silently keeping the library default for that term.

Validation: benchmarks/opsd/test_rollout_prefill.py — 3 passed. The regression covers an absent cost map and each of the five missing keys individually, and asserts the sweep is not run in the rejected cases. py_compile, diff --check and flake8 (no new findings) pass. Repository pre-commit hooks exclude benchmarks/opsd, so the direct checks were used.

Companion DeepSpeed PR #8786 was refreshed to current master and is mergeable; its full rollout/cache suite passes on an H20Z GPU (159 passed, 3 skipped).

banxingmjj pushed a commit to openanolis/DeepSpeed that referenced this pull request Nov 15, 2026
…ost model (deepspeedai#8786)

## November 11 update

Rebased onto master `b9100c5`, which includes deepspeedai#8390 and deepspeedai#9008; new head
`82ac56e`.

Uses deepspeedai#8939's shared sampling kwargs, token selector and FP32
repetition-penalty helper. Manual graph and continuous-batching paths
retain upstream temperature/top-k/top-p sampling. Adaptive prefill
retains its single-sample greedy scope, including fixed-slot graph
decode. deepspeedai#9004's pre-built 4-D graph mask is preserved.

- **deepspeedai#8390 early stopping:** ordinary generation and Auto's batched
fallback share deepspeedai#8390's generation kwargs, so they stop on natural EOS;
ZeRO-3 keeps the group-aware synchronized stop. Auto remains
single-process.
- **Review fixes:** the graph path rejects `align_decode_fronts=True`
only when Auto is disabled, because Auto always uses aligned bucketed
prefill. The accelerator graph test covers both flag values. The
documentation states that Auto takes precedence over
`align_decode_fronts` on the graph path; cache trimming is still
rejected.
- **deepspeedai#9008 lazy KV layers:** the bucketed prefill copy initializes
uninitialized cache layers through the shared
`_continuous_cache_layer()` helper, like the equal-width path.
- **Codex P2 (preparer findings):** after a custom
`_prepare_generation_config()` runs, Auto rejects the prepared config if
it changes any request override (token budget, EOS, PAD or greedy
settings). Otherwise the batched route would honor the change inside
`generate()` while the CB routes keep the request values. A preparer
that replaces or edits the attention mask is rejected for the same
reason.
- **Codex P2 (mask lengths):** effective prompt lengths use
`count_nonzero`, so BF16/FP16 masks longer than their exact-integer
range no longer round the bucket width.
- **Codex P1 (test reliability):** the route-planning profile test uses
a frozen clock advanced by the model's generation-config preparer,
instead of wall-clock bounds or mocked rollout internals.
- The decode cost still uses the configured maximum generation budget.
Natural EOS can shorten decode, so the extra-CB estimate can be too
large; the documentation states this assumption.

Validation on an 8 × H200 node, PyTorch 2.9.1+cu128, CUDA 12.8,
Transformers 4.57.1: **232 passed, 3 skipped, 2 deselected** across the
runtime rollout, scheduler, interface, static-cache and v1 accelerator
suites, including deepspeedai#8390's 4-rank subgroup and 2-GPU ZeRO-3
synchronized-stop tests. Skips require the newer cache query-offset API;
deselections are the native shared-prefill kernel and optional
pretrained-model test. Pre-commit passed on all changed files. A small
natural-EOS check (Qwen2.5-3B-Instruct, four short questions) ended
every response at EOS, with Auto matching ordinary generation. This
update used AI assistance.

The performance tables below retain their original revision attribution;
large-model throughput and peak memory were not remeasured for this
rebase.

## Summary

Closes deepspeedai#8785. Related tracking:
[deepspeedai#8197](deepspeedai#8197),
[deepspeedai#8462](deepspeedai#8462).

**Performance follow-up to
[deepspeedai#8497](deepspeedai#8497 Aligned
CB avoids padding on long-tail prompts, but one prefill Forward per
request is expensive for short prompts. A padding-ratio threshold cannot
distinguish **128/4** from **512/16**, despite identical ratios and
different fastest baselines.

This change adds **one opt-in automatic policy** that compares ordinary
no-CB generation with aligned CB using **DP-selected prefill buckets**.
For 1×512 + 31×16, the selected buckets are `[1,512]` and `[31,16]`,
reducing prefill Forwards from 32 to 2.

**Auto is disabled by default.** Existing manual modes and positional
configuration arguments are preserved.

```python
cfg = HybridEngineRolloutConfig(adaptive_prefill=True)
rollout = HybridEngineRollout(engine, tokenizer, cfg)
sampling = SamplingConfig(max_new_tokens=16, temperature=0, continuous_batch_size=32)
```

## Selection and configuration

A length-sorted **O(n²)-time / O(n)-space DP** minimizes the additive
estimated cost of contiguous buckets. It explores one batch, singleton
batches and intermediate partitions without running the GPU for
candidate evaluation. Each bucket uses ordinary left padding and a 2D
attention mask; **no varlen kernel is needed**. Model geometry is read
once per plan.

```text
C(B,L) = fixed + token × B×L + attention × estimated_attention_GFLOPs
         + kv × estimated_KV_MiB
attention_GFLOPs = 4 × B × L² × layers × query_heads × head_dim / 1e9
KV_MiB = 2 × B × L × layers × KV_heads × head_dim × element_bytes / 2^20
```

| Parameter | Default | Meaning |
| --- | ---: | --- |
| `prefill_fixed_cost_ms` | 25.0 | Fixed milliseconds per prefill
Forward |
| `prefill_token_cost_ms` | 0.1 | Milliseconds per padded token
position; projection/MLP term |
| `prefill_attention_cost_ms` | 0.01 | Milliseconds per estimated
attention GFLOP |
| `prefill_kv_cost_ms` | 0.04 | Milliseconds per MiB of KV copied into
the CB cache |
| `continuous_decode_cost_ms` | 10.0 | Additional CB milliseconds per
remaining Decode step |
| `prefill_max_tokens` | 65536 | Maximum padded token positions per
prefill Forward; `None` removes this limit |

All cost coefficients must be finite and non-negative. **These are
starting estimates, not universal GPU constants**; calibrate them for
the model, device, precision and attention backend. Head dimension,
query/KV heads, layers and parameter element size are read
automatically. The DP is optimal for the configured estimates, not
guaranteed globally optimal measured latency.

Ordinary generation is eligible when every row fits
`continuous_batch_size`, the physical input fits the prefill/context
limits, and neither explicit static capacity nor trimming is requested.
Its estimated cost omits CB cache-copy traffic; CB adds
`continuous_decode_cost_ms × (max_new_tokens - 1)`. The first output
token comes from Prefill.

For CB, **every admission/refill replans the newly admitted requests**.
Effective KV spans are copied into the existing right-aligned cache;
surviving rows retain their layout. The policy does not change caller
configuration or active-row bounds. Disabling Auto restores manual
selection. Model/OOM errors propagate without trying another path.

## Correctness

- Apply repetition penalty from the first token using **original padded
history**, even when bucket model inputs are trimmed.
- Preserve generated PAD-valued tokens through the first EOS, excluding
structural padding after termination.
- Include route planning and result normalization in end-to-end
profiling; count **productive output tokens** and retain
active/configured capacity fields.
- Resolve HF's effective generation configuration before rejecting
unsupported processors/output modes, including legacy model-config
refresh; preserve explicit generation-config overrides. Auto requires
single-process greedy rollout, one sample per prompt and cache-class
support. With graph capture, Auto uses eager bucketed prefill and
fixed-slot graph decode; cache trimming and shared prefill remain
unsupported.

## Experiments

Related work: [TurboTransformers, §5 (PPoPP
2021)](https://arxiv.org/html/2010.05680) already uses length sorting,
profiled batch costs and an O(n²) DP to partition variable-length
inference requests. [DynaPipe, §4](https://arxiv.org/html/2311.10418)
applies cost-based DP partitioning to training micro-batches. This PR
applies that established scheduling approach to HybridEngine rollout,
with ordinary-generation comparison, refill planning and existing
aligned-cache semantics; it makes no claim of a new DP algorithm.

**Single GPU; Qwen/Qwen2.5-3B-Instruct; FP32 / SDPA; PyTorch
2.9.1+cu128; Transformers 4.57.1.** Each candidate receives **one warmup
and five interleaved timed runs**, 32 requests and 16 output tokens per
request. CB capacity is 32; trimming is disabled, cache capacity unset,
TF32 disabled. The default cost coefficients above are unchanged across
workloads.

| Workload | Effective prompt lengths | Reason |
| --- | --- | --- |
| Short, similar | 20..13 repeated four times | Normal workload shape
from deepspeedai#8497 |
| Long tail | 1×512 + 31×16 | Long-tail workload from deepspeedai#8497 |
| Short tail | 1×20 + 31×1 | Counterexample to ratio-only routing |
| Moderate tail | 1×128 + 31×4 | Same padding ratio as 512/16, different
best legacy path |
| Spread | 16, 32, …, 512 | General distribution with intermediate
buckets |

Inputs repeat non-special token ID 14990. deepspeedai#8497 did not publish complete
short-prompt sequences, so that case reconstructs its documented shape.
Every timed response matches independent single-request eager
generation; each workload produces **512 effective output tokens**.

GPU-synchronized wall-clock timing includes the full `generate()` call,
including Auto planning and normalization. Model loading and oracle work
are outside timing; profiling is disabled during timed runs. All
reported E2E values are **arithmetic means of five runs**.

| Workload | No-CB (ms) | Equal-width CB (ms) | Aligned CB (ms) | Auto
(ms) | Auto / best | Auto choice |
| --- | ---: | ---: | ---: | ---: | ---: | --- |
| Short, similar | **505.54** | 673.66 | 1549.74 | **501.77** | 0.9925×
| No-CB |
| 1×512 + 31×16 | 2458.46 | 2596.38 | **1682.10** | **840.06** | 0.4994×
| 2 buckets |
| 1×20 + 31×1 | **501.74** | 667.57 | 1331.82 | **497.57** | 0.9917× |
No-CB |
| 1×128 + 31×4 | **897.17** | 1053.26 | 1468.58 | **634.59** | 0.7073× |
2 buckets |
| 16, 32, …, 512 | 2458.13 | 2595.33 | **2337.75** | **1984.86** |
0.8490× | 6 buckets |

**All five workloads meet the 5% acceptance tolerance.** Auto chooses
ordinary generation for both short cases; their small timing differences
are not claimed as a kernel speedup. Relative to the fastest legacy
baseline, mean latency falls by **50.06%** for 1×512 + 31×16, **29.27%**
for 1×128 + 31×4, **15.10%** for 16, 32, …, 512. These improvements come
from fewer prefill Forwards and reduced padded work.

<details>
<summary>Throughput and peak allocated memory for every path</summary>

Throughput is 512 divided by mean latency. Memory is peak PyTorch
allocated memory during timed calls, including model weights.

| Workload | Path | Output tok/s | Peak allocated memory (MiB) |
| --- | --- | ---: | ---: |
| Short, similar | No-CB | 1012.77 | 11988.96 |
| Short, similar | Equal-width CB | 760.03 | 12036.75 |
| Short, similar | Aligned CB | 330.38 | 11928.77 |
| Short, similar | **Auto** | 1020.40 | 11988.96 |
| 1×512 + 31×16 | No-CB | 208.26 | 15558.69 |
| 1×512 + 31×16 | Equal-width CB | 197.20 | 16747.20 |
| 1×512 + 31×16 | Aligned CB | 304.38 | 13278.51 |
| 1×512 + 31×16 | **Auto** | 609.48 | 13278.49 |
| 1×20 + 31×1 | No-CB | 1020.46 | 11994.64 |
| 1×20 + 31×1 | Equal-width CB | 766.96 | 12033.50 |
| 1×20 + 31×1 | Aligned CB | 384.44 | 11925.07 |
| 1×20 + 31×1 | **Auto** | 1029.00 | 11994.64 |
| 1×128 + 31×4 | No-CB | 570.68 | 12742.62 |
| 1×128 + 31×4 | Equal-width CB | 486.11 | 13066.75 |
| 1×128 + 31×4 | Aligned CB | 348.64 | 12221.97 |
| 1×128 + 31×4 | **Auto** | 806.82 | 12221.95 |
| 16, 32, …, 512 | No-CB | 208.29 | 15558.69 |
| 16, 32, …, 512 | Equal-width CB | 197.28 | 16747.20 |
| 16, 32, …, 512 | Aligned CB | 219.01 | 13278.56 |
| 16, 32, …, 512 | **Auto** | 257.95 | 13682.33 |

</details>

<details>
<summary>Separate profiling breakdown</summary>

One additional profiling call per candidate, outside acceptance timing.
These diagnostics are not averages. Cache management is part of the
generation overhead, not an additional independent total.

| Workload | Path | Prefill (ms) | Decode (ms) | Cache management (ms) |
Prefill / Decode Forwards |
| --- | --- | ---: | ---: | ---: | ---: |
| Short, similar | No-CB | 91.01 | 401.30 | N/A | 1 / 15 |
| Short, similar | Equal-width CB | 91.99 | 499.53 | 26.80 | 1 / 15 |
| Short, similar | Aligned CB | 955.52 | 500.30 | 35.25 | 32 / 15 |
| Short, similar | **Auto** | 90.74 | 396.90 | N/A | 1 / 15 |
| 1×512 + 31×16 | No-CB | 1918.22 | 531.89 | N/A | 1 / 15 |
| 1×512 + 31×16 | Equal-width CB | 1919.76 | 595.14 | 27.09 | 1 / 15 |
| 1×512 + 31×16 | Aligned CB | 981.04 | 594.52 | 35.02 | 32 / 15 |
| 1×512 + 31×16 | **Auto** | 153.66 | 592.66 | 31.43 | 2 / 15 |
| 1×20 + 31×1 | No-CB | 91.06 | 402.45 | N/A | 1 / 15 |
| 1×20 + 31×1 | Equal-width CB | 91.84 | 494.84 | 26.88 | 1 / 15 |
| 1×20 + 31×1 | Aligned CB | 743.30 | 491.93 | 34.48 | 32 / 15 |
| 1×20 + 31×1 | **Auto** | 90.62 | 394.78 | N/A | 1 / 15 |
| 1×128 + 31×4 | No-CB | 490.08 | 399.13 | N/A | 1 / 15 |
| 1×128 + 31×4 | Equal-width CB | 490.85 | 482.08 | 27.09 | 1 / 15 |
| 1×128 + 31×4 | Aligned CB | 889.01 | 483.21 | 34.83 | 32 / 15 |
| 1×128 + 31×4 | **Auto** | 63.80 | 485.00 | 31.43 | 2 / 15 |
| 16, 32, …, 512 | No-CB | 1918.03 | 531.64 | N/A | 1 / 15 |
| 16, 32, …, 512 | Equal-width CB | 1918.95 | 594.68 | 27.32 | 1 / 15 |
| 16, 32, …, 512 | Aligned CB | 1648.16 | 593.25 | 35.91 | 32 / 15 |
| 16, 32, …, 512 | **Auto** | 1301.59 | 593.31 | 32.49 | 6 / 15 |

The 128/4 and 512/16 workloads reduce prefill Forwards from 32 on
aligned CB to **2 on Auto**. Spread uses **6 buckets**. All CB paths in
these performance runs have zero trims and zero moved bytes; ordinary
generation has no static-cache statistics. This makes no new
trimming-speedup claim.

</details>

## Validation and limits

A separate microbenchmark measures **planner overhead**, without model
Prefill: for 128/256 requests of 65,536 tokens, CPU DP takes **0.48/1.82
ms** on average with the default per-bucket token limit. It still checks
all 8,256/32,896 candidate segments; over-limit segments skip cost
evaluation. For 32 requests, DP takes 0.089 ms at length512 and 0.040 ms
at length65,536 (0.078 ms with no token limit). GPU mask
validation/length extraction for 32×65,536 takes 0.83 ms separately.
Measurements use ten repeated timing samples after warmup. **These are
not 64k model Prefill timings or the complete Auto-path overhead.**

**The original validation passed 105 related rollout/cache tests**,
including independent exhaustive partition comparisons for DP,
device-backed tiny-Qwen tests for grouped prefill and refill, both cache
APIs, effective generation-config/legacy refresh, repetition history,
EOS/PAD masks, capacity bounds and external-wall-time checks for
profiling. Existing tests cover retirement, compaction and trimming.
Independent mixed-bucket checks also exercised live-survivor refill and
trimming on/off against padded per-request HF generation.

One pre-existing native shared-prefill extension test is excluded
because the host has no CUDA compiler; **all added tests ran**.
Changed-file pre-commit and `git diff --check` pass. Repeated local
review ended with **no actionable findings**.

```sh
python -m pytest -q \
  tests/unit/runtime/rollout/test_hybrid_engine_rollout.py \
  tests/unit/runtime/rollout/test_continuous_batching.py \
  tests/unit/runtime/rollout/test_rollout_interface.py \
  tests/unit/utils/test_static_cache.py \
  -k 'not native_repeat_kv_cache_fp16_reverse_copy'
```

The prefill token limit does **not** bound allocated decode KV memory. A
single prompt over that limit raises an explicit error; there is no
chunked prefill. **64k prompts require a model context limit covering
prompt plus output budget, sufficient KV capacity and temporary
memory.** The measured checkpoint is configured for 32k.
Planner/position-limit boundaries are tested; real 64k inference
performance is not claimed.

Performance evidence uses unsharded HF models behind
`HybridEngineRollout`; the additional calibrated and real-question
checks are listed below. Other model/device configurations, FP16, native
kernel injection, ZeRO/TP training integration and production
arrival/EOS distributions remain unmeasured. Controlled performance runs
do not exercise staggered EOS or trimming; those have correctness
coverage.

No new dependency, kernel, online learner or trimming redesign. Related
foundation: [deepspeedai#8368](deepspeedai#8368);
profiling: [deepspeedai#8494](deepspeedai#8494).

## Reviewer follow-up

[Commit
65a9115](deepspeedai@65a9115)
propagates the resolved generation configuration through continuous
Prefill, Decode and refill, and omits `output_logits` from legacy manual
generation. **27 targeted tests pass**, including six new cases covering
both Auto routes, both cache APIs and manual equal-width/aligned
compatibility. The new tests reproduced four failures before the fix.
The compatibility test uses a strict generation wrapper that rejects the
unsupported keyword.

Calibration and evaluation tooling moved to [DeepSpeedExamples
deepspeedai#1018](deepspeedai/DeepSpeedExamples#1018), with
its fitting regression test and usage documentation. **Merge the
Examples companion before this PR so the documentation link is
available.** The Examples companion includes MHA-head fallback and
validates a source fingerprint covering rollout, planning and
static-cache code. A seeded tiny-model GPU smoke check completed
calibration and all four saved-config sweep workloads with matching
output IDs on every path. Two independent local reviews returned no
findings.

The calibrated sweep from the preceding routing-overhead fix is separate
from the original 16-token experiments above. It uses **two EOS-disabled
output tokens**, SDPA and one GPU, with independent uniform calibration
shapes and one unchanged coefficient set per configuration:

| Configuration | 128×16 | 1×128 + 31×4 | 16..512 | 1×512 + 31×16 |
Worst regression |
| --- | ---: | ---: | ---: | ---: | ---: |
| Tiny Qwen2 / FP32 | 1.0220× | 0.9685× | 0.9539× | 0.9647× | +2.205% |
| Qwen2.5-3B / FP32 | 1.0005× | 0.2345× | 0.7808× | 0.2265× | +0.054% |
| Qwen3-32B / BF16 | 1.0027× | 0.3881× | 0.7189× | 0.1811× | +0.273% |

Values are **Auto E2E / fastest manual baseline E2E**. The two larger
models select no-CB for 128×16 and bucketed CB for the other three
cases; the tiny model selects no-CB throughout. Auto matches ordinary
output IDs in all twelve cases. Target-model means use three interleaved
repetitions after warmup; the tiny test uses ten. [Full sweep details
and calibration
limits](deepspeedai#8786 (comment)).

A separate [Qwen3-32B real-question
check](deepspeedai#8786 (comment))
selected no-CB on both workloads: MATH-500 averaged **31.452 s / 31.393
s** for ordinary/Auto over three runs; ARC-AGI-2 measured **237.560 s /
237.888 s** in one paired run. All eight responses matched. These checks
cover complete generation calls, excluding tokenization, loading,
rewards and training.



A real tiny-OPT calibration additionally exposed a missing HF cache
capability flag. [Commit
ecacbce](deepspeedai@ecacbce)
adds `is_compileable=True` to the static-cache container, matching its
existing static layers. **Six targeted GPU checks pass**, including new
OPT manual/Auto comparisons with padded prompts and refill against the
HF oracle, plus existing Qwen bucket/cache-path checks. The three
Examples regression tests cover fitting, complete OPT calibration and
saved-source compatibility. Both local reviews found no issues in this
follow-up.

## Final correctness and performance acceptance

[Commit
d4def3d](deepspeedai@d4def3d)
includes the final fixes: non-default repetition penalties are
calculated in FP32 like HF, and unknown public generation switches are
rejected before routing. The default-penalty path retains its early
return. **Five focused BF16 checks and six unknown-setting CPU cases
pass.** The Examples CLI now requires complete metadata for saved
calibration files. Both independent correctness and performance reviews
returned no findings after these fixes.

A new calibration and sweep were run against measured commit `d4def3d`.
**Qwen3-32B, BF16, SDPA, one GPU; two EOS-disabled output tokens; one
warmup plus three interleaved timed runs per path.** Calibration and
held-out evaluation use separate shapes. The calibration file, sweep and
that measured revision's rollout/planner/static-cache sources have the
same combined SHA256:
`f9d79a43d2e6c0e3fef164ef11e81a26c643820beb1a0d3f1a86f41a46dbdd0c`.

| Workload | No-CB ms | Equal-width CB ms | Aligned CB ms | Auto ms |
Auto / best | Auto path |
| --- | ---: | ---: | ---: | ---: | ---: | --- |
| 128×16 | **269.83** | 439.91 | 5717.89 | 270.03 | 1.000743× | No-CB |
| 1×128 + 31×4 | **496.15** | 540.76 | 1501.57 | **191.96** | 0.386886×
| 2 buckets |
| 16,32,…,512 | 1962.87 | 2009.18 | **1864.55** | **1336.57** |
0.716828× | 6 buckets |
| 1×512 + 31×16 | 1962.56 | 2005.33 | **1575.08** | **288.31** |
0.183045× | 2 buckets |

**All four pass the 5% acceptance tolerance; worst measured regression
is +0.07428%. Auto output IDs match ordinary generation in every case**,
and both routes are exercised. Means and ratios were independently
recomputed from raw samples. The legacy aligned spread baseline retains
its previously recorded second-token difference on one row; Auto matches
no-CB there. Fixed output length prevents this numerical-sensitive
baseline difference from changing the amount of timed work.

The fitted coefficients are fixed=28.791679 ms, token=0.093362064 ms,
attention=0.010090454 ms/GFLOP, KV=0.056288540 ms/MiB and additional
decode=17.144541 ms/step. The earlier tables remain historical
measurements on their original versions. This final check does not
establish long-context, natural-EOS, 256-request E2E or dynamic-arrival
performance. Auto remains disabled by default.


[Commit
a0588b7](deepspeedai@a0588b7)
then fixes manual aligned prefill to retain the original padded
repetition history. Its focused GPU regression reproduced the mismatch
and now passes. Auto already passed this same mapping, and the measured
default-penalty paths do not read it; independent correctness and
performance delta reviews returned no findings. The table remains
attributed to `d4def3d`, with no timings or hashes relabeled as a
new-HEAD measurement. The Examples benchmark explicitly requires
Transformers >=4.56; the library's existing compatibility scope is
unchanged.

## Review invitation

cc @nathon-lee @delock — this follows your continuous-batching,
profiling, and aligned-cache work in deepspeedai#8368, deepspeedai#8494, and deepspeedai#8497. Could you
review the automatic route selection, bucket/refill cache semantics, and
cost defaults?


## November refresh

Rebased onto `b62afa8` and squashed to one commit; the PR is now
mergeable. The refresh keeps deepspeedai#8732 graph state, deepspeedai#8839 last-position
prefill logits, deepspeedai#8933 accelerator-neutral graph calls and deepspeedai#8852
`DeepSpeedStaticCache.get_query_offset()`. deepspeedai#8980's explicit accelerator
opt-in is respected: a backend without `is_graph_capture_supported()`
falls back to eager before any routing decision.

*Superseded by 8a9641b:* Auto now combines eager bucketed prefill with
fixed-slot graph decode (see the November 11 update). SegKI (deepspeedai#8602) is
not part of this PR.

Validation, 8×H200-class node with one GPU in use (NVIDIA H20Z, PyTorch
2.9.1+cu128, CUDA 12.8, Transformers 4.57.1):

| Check | Result |
| --- | --- |
| rollout/cache suite | 159 passed, 3 skipped, 1 deselected |
| Skips | Three Transformers 5.20-only query-offset cases added by deepspeedai#8852
|
| Deselected | Native shared-prefill kernel test; the test host has no
CUDA compiler |
| Changed-file pre-commit | Passed |

Graph replay, resolved-config propagation on both routes (FP32/BF16), DP
bucket/refill oracles, Auto/Graph rejection (since replaced by joint
routing) and manual graph dispatch all pass. No new performance numbers
are claimed: the tables above were measured before deepspeedai#8839/deepspeedai#8732, so
refreshed baselines need a separate calibration run if wanted.

AI assistance was used for the rebase, conflict resolution, test
execution and drafting of this description; the submitting human
reviewed the diff and the results above.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: SuperGoodGame <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant