Repository navigation
benchmarks: move adaptive prefill calibration to Examples - #1018
SuperGoodGame wants to merge 5 commits into
Conversation
Signed-off-by: SuperGoodGame <[email protected]>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dcdde0bf7f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: SuperGoodGame <[email protected]>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 79b2140eaf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: SuperGoodGame <[email protected]>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: beb3401dce
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: SuperGoodGame <[email protected]>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3fd685fc55
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| 'allow_tf32', 'source_sha256'): | ||
| if calibration['metadata'].get(key) != metadata[key]: | ||
| parser.error(f'calibration differs in {key}; recalibrate for this configuration') | ||
| costs = calibration.get('costs', {}) |
There was a problem hiding this comment.
Reject incomplete saved cost maps
When --cost-config points to an otherwise matching calibration artifact whose costs member is absent or has lost one of the five emitted coefficients, this fallback silently supplies HybridEngineRolloutConfig defaults for the missing values and the sweep completes with current metadata as though the calibrated policy was evaluated. Since --calibrate always emits the complete map (and hand-tuned partial configurations are documented for direct application use), validate the required cost keys before running the sweep.
Useful? React with 👍 / 👎.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
A supplied calibration file must provide all five cost coefficients. A missing cost map, or any missing coefficient, now fails the CLI instead of silently falling back to defaults for that term. The CLI regression checks a missing cost map and each missing key. Signed-off-by: SuperGoodGame <[email protected]>
ba438e5 to
8c73cbd
Compare
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Added Validation: Companion DeepSpeed PR #8786 was refreshed to current master and is mergeable; its full rollout/cache suite passes on an H20Z GPU (159 passed, 3 skipped). |
…ost model (deepspeedai#8786) ## November 11 update Rebased onto master `b9100c5`, which includes deepspeedai#8390 and deepspeedai#9008; new head `82ac56e`. Uses deepspeedai#8939's shared sampling kwargs, token selector and FP32 repetition-penalty helper. Manual graph and continuous-batching paths retain upstream temperature/top-k/top-p sampling. Adaptive prefill retains its single-sample greedy scope, including fixed-slot graph decode. deepspeedai#9004's pre-built 4-D graph mask is preserved. - **deepspeedai#8390 early stopping:** ordinary generation and Auto's batched fallback share deepspeedai#8390's generation kwargs, so they stop on natural EOS; ZeRO-3 keeps the group-aware synchronized stop. Auto remains single-process. - **Review fixes:** the graph path rejects `align_decode_fronts=True` only when Auto is disabled, because Auto always uses aligned bucketed prefill. The accelerator graph test covers both flag values. The documentation states that Auto takes precedence over `align_decode_fronts` on the graph path; cache trimming is still rejected. - **deepspeedai#9008 lazy KV layers:** the bucketed prefill copy initializes uninitialized cache layers through the shared `_continuous_cache_layer()` helper, like the equal-width path. - **Codex P2 (preparer findings):** after a custom `_prepare_generation_config()` runs, Auto rejects the prepared config if it changes any request override (token budget, EOS, PAD or greedy settings). Otherwise the batched route would honor the change inside `generate()` while the CB routes keep the request values. A preparer that replaces or edits the attention mask is rejected for the same reason. - **Codex P2 (mask lengths):** effective prompt lengths use `count_nonzero`, so BF16/FP16 masks longer than their exact-integer range no longer round the bucket width. - **Codex P1 (test reliability):** the route-planning profile test uses a frozen clock advanced by the model's generation-config preparer, instead of wall-clock bounds or mocked rollout internals. - The decode cost still uses the configured maximum generation budget. Natural EOS can shorten decode, so the extra-CB estimate can be too large; the documentation states this assumption. Validation on an 8 × H200 node, PyTorch 2.9.1+cu128, CUDA 12.8, Transformers 4.57.1: **232 passed, 3 skipped, 2 deselected** across the runtime rollout, scheduler, interface, static-cache and v1 accelerator suites, including deepspeedai#8390's 4-rank subgroup and 2-GPU ZeRO-3 synchronized-stop tests. Skips require the newer cache query-offset API; deselections are the native shared-prefill kernel and optional pretrained-model test. Pre-commit passed on all changed files. A small natural-EOS check (Qwen2.5-3B-Instruct, four short questions) ended every response at EOS, with Auto matching ordinary generation. This update used AI assistance. The performance tables below retain their original revision attribution; large-model throughput and peak memory were not remeasured for this rebase. ## Summary Closes deepspeedai#8785. Related tracking: [deepspeedai#8197](deepspeedai#8197), [deepspeedai#8462](deepspeedai#8462). **Performance follow-up to [deepspeedai#8497](deepspeedai#8497 Aligned CB avoids padding on long-tail prompts, but one prefill Forward per request is expensive for short prompts. A padding-ratio threshold cannot distinguish **128/4** from **512/16**, despite identical ratios and different fastest baselines. This change adds **one opt-in automatic policy** that compares ordinary no-CB generation with aligned CB using **DP-selected prefill buckets**. For 1×512 + 31×16, the selected buckets are `[1,512]` and `[31,16]`, reducing prefill Forwards from 32 to 2. **Auto is disabled by default.** Existing manual modes and positional configuration arguments are preserved. ```python cfg = HybridEngineRolloutConfig(adaptive_prefill=True) rollout = HybridEngineRollout(engine, tokenizer, cfg) sampling = SamplingConfig(max_new_tokens=16, temperature=0, continuous_batch_size=32) ``` ## Selection and configuration A length-sorted **O(n²)-time / O(n)-space DP** minimizes the additive estimated cost of contiguous buckets. It explores one batch, singleton batches and intermediate partitions without running the GPU for candidate evaluation. Each bucket uses ordinary left padding and a 2D attention mask; **no varlen kernel is needed**. Model geometry is read once per plan. ```text C(B,L) = fixed + token × B×L + attention × estimated_attention_GFLOPs + kv × estimated_KV_MiB attention_GFLOPs = 4 × B × L² × layers × query_heads × head_dim / 1e9 KV_MiB = 2 × B × L × layers × KV_heads × head_dim × element_bytes / 2^20 ``` | Parameter | Default | Meaning | | --- | ---: | --- | | `prefill_fixed_cost_ms` | 25.0 | Fixed milliseconds per prefill Forward | | `prefill_token_cost_ms` | 0.1 | Milliseconds per padded token position; projection/MLP term | | `prefill_attention_cost_ms` | 0.01 | Milliseconds per estimated attention GFLOP | | `prefill_kv_cost_ms` | 0.04 | Milliseconds per MiB of KV copied into the CB cache | | `continuous_decode_cost_ms` | 10.0 | Additional CB milliseconds per remaining Decode step | | `prefill_max_tokens` | 65536 | Maximum padded token positions per prefill Forward; `None` removes this limit | All cost coefficients must be finite and non-negative. **These are starting estimates, not universal GPU constants**; calibrate them for the model, device, precision and attention backend. Head dimension, query/KV heads, layers and parameter element size are read automatically. The DP is optimal for the configured estimates, not guaranteed globally optimal measured latency. Ordinary generation is eligible when every row fits `continuous_batch_size`, the physical input fits the prefill/context limits, and neither explicit static capacity nor trimming is requested. Its estimated cost omits CB cache-copy traffic; CB adds `continuous_decode_cost_ms × (max_new_tokens - 1)`. The first output token comes from Prefill. For CB, **every admission/refill replans the newly admitted requests**. Effective KV spans are copied into the existing right-aligned cache; surviving rows retain their layout. The policy does not change caller configuration or active-row bounds. Disabling Auto restores manual selection. Model/OOM errors propagate without trying another path. ## Correctness - Apply repetition penalty from the first token using **original padded history**, even when bucket model inputs are trimmed. - Preserve generated PAD-valued tokens through the first EOS, excluding structural padding after termination. - Include route planning and result normalization in end-to-end profiling; count **productive output tokens** and retain active/configured capacity fields. - Resolve HF's effective generation configuration before rejecting unsupported processors/output modes, including legacy model-config refresh; preserve explicit generation-config overrides. Auto requires single-process greedy rollout, one sample per prompt and cache-class support. With graph capture, Auto uses eager bucketed prefill and fixed-slot graph decode; cache trimming and shared prefill remain unsupported. ## Experiments Related work: [TurboTransformers, §5 (PPoPP 2021)](https://arxiv.org/html/2010.05680) already uses length sorting, profiled batch costs and an O(n²) DP to partition variable-length inference requests. [DynaPipe, §4](https://arxiv.org/html/2311.10418) applies cost-based DP partitioning to training micro-batches. This PR applies that established scheduling approach to HybridEngine rollout, with ordinary-generation comparison, refill planning and existing aligned-cache semantics; it makes no claim of a new DP algorithm. **Single GPU; Qwen/Qwen2.5-3B-Instruct; FP32 / SDPA; PyTorch 2.9.1+cu128; Transformers 4.57.1.** Each candidate receives **one warmup and five interleaved timed runs**, 32 requests and 16 output tokens per request. CB capacity is 32; trimming is disabled, cache capacity unset, TF32 disabled. The default cost coefficients above are unchanged across workloads. | Workload | Effective prompt lengths | Reason | | --- | --- | --- | | Short, similar | 20..13 repeated four times | Normal workload shape from deepspeedai#8497 | | Long tail | 1×512 + 31×16 | Long-tail workload from deepspeedai#8497 | | Short tail | 1×20 + 31×1 | Counterexample to ratio-only routing | | Moderate tail | 1×128 + 31×4 | Same padding ratio as 512/16, different best legacy path | | Spread | 16, 32, …, 512 | General distribution with intermediate buckets | Inputs repeat non-special token ID 14990. deepspeedai#8497 did not publish complete short-prompt sequences, so that case reconstructs its documented shape. Every timed response matches independent single-request eager generation; each workload produces **512 effective output tokens**. GPU-synchronized wall-clock timing includes the full `generate()` call, including Auto planning and normalization. Model loading and oracle work are outside timing; profiling is disabled during timed runs. All reported E2E values are **arithmetic means of five runs**. | Workload | No-CB (ms) | Equal-width CB (ms) | Aligned CB (ms) | Auto (ms) | Auto / best | Auto choice | | --- | ---: | ---: | ---: | ---: | ---: | --- | | Short, similar | **505.54** | 673.66 | 1549.74 | **501.77** | 0.9925× | No-CB | | 1×512 + 31×16 | 2458.46 | 2596.38 | **1682.10** | **840.06** | 0.4994× | 2 buckets | | 1×20 + 31×1 | **501.74** | 667.57 | 1331.82 | **497.57** | 0.9917× | No-CB | | 1×128 + 31×4 | **897.17** | 1053.26 | 1468.58 | **634.59** | 0.7073× | 2 buckets | | 16, 32, …, 512 | 2458.13 | 2595.33 | **2337.75** | **1984.86** | 0.8490× | 6 buckets | **All five workloads meet the 5% acceptance tolerance.** Auto chooses ordinary generation for both short cases; their small timing differences are not claimed as a kernel speedup. Relative to the fastest legacy baseline, mean latency falls by **50.06%** for 1×512 + 31×16, **29.27%** for 1×128 + 31×4, **15.10%** for 16, 32, …, 512. These improvements come from fewer prefill Forwards and reduced padded work. <details> <summary>Throughput and peak allocated memory for every path</summary> Throughput is 512 divided by mean latency. Memory is peak PyTorch allocated memory during timed calls, including model weights. | Workload | Path | Output tok/s | Peak allocated memory (MiB) | | --- | --- | ---: | ---: | | Short, similar | No-CB | 1012.77 | 11988.96 | | Short, similar | Equal-width CB | 760.03 | 12036.75 | | Short, similar | Aligned CB | 330.38 | 11928.77 | | Short, similar | **Auto** | 1020.40 | 11988.96 | | 1×512 + 31×16 | No-CB | 208.26 | 15558.69 | | 1×512 + 31×16 | Equal-width CB | 197.20 | 16747.20 | | 1×512 + 31×16 | Aligned CB | 304.38 | 13278.51 | | 1×512 + 31×16 | **Auto** | 609.48 | 13278.49 | | 1×20 + 31×1 | No-CB | 1020.46 | 11994.64 | | 1×20 + 31×1 | Equal-width CB | 766.96 | 12033.50 | | 1×20 + 31×1 | Aligned CB | 384.44 | 11925.07 | | 1×20 + 31×1 | **Auto** | 1029.00 | 11994.64 | | 1×128 + 31×4 | No-CB | 570.68 | 12742.62 | | 1×128 + 31×4 | Equal-width CB | 486.11 | 13066.75 | | 1×128 + 31×4 | Aligned CB | 348.64 | 12221.97 | | 1×128 + 31×4 | **Auto** | 806.82 | 12221.95 | | 16, 32, …, 512 | No-CB | 208.29 | 15558.69 | | 16, 32, …, 512 | Equal-width CB | 197.28 | 16747.20 | | 16, 32, …, 512 | Aligned CB | 219.01 | 13278.56 | | 16, 32, …, 512 | **Auto** | 257.95 | 13682.33 | </details> <details> <summary>Separate profiling breakdown</summary> One additional profiling call per candidate, outside acceptance timing. These diagnostics are not averages. Cache management is part of the generation overhead, not an additional independent total. | Workload | Path | Prefill (ms) | Decode (ms) | Cache management (ms) | Prefill / Decode Forwards | | --- | --- | ---: | ---: | ---: | ---: | | Short, similar | No-CB | 91.01 | 401.30 | N/A | 1 / 15 | | Short, similar | Equal-width CB | 91.99 | 499.53 | 26.80 | 1 / 15 | | Short, similar | Aligned CB | 955.52 | 500.30 | 35.25 | 32 / 15 | | Short, similar | **Auto** | 90.74 | 396.90 | N/A | 1 / 15 | | 1×512 + 31×16 | No-CB | 1918.22 | 531.89 | N/A | 1 / 15 | | 1×512 + 31×16 | Equal-width CB | 1919.76 | 595.14 | 27.09 | 1 / 15 | | 1×512 + 31×16 | Aligned CB | 981.04 | 594.52 | 35.02 | 32 / 15 | | 1×512 + 31×16 | **Auto** | 153.66 | 592.66 | 31.43 | 2 / 15 | | 1×20 + 31×1 | No-CB | 91.06 | 402.45 | N/A | 1 / 15 | | 1×20 + 31×1 | Equal-width CB | 91.84 | 494.84 | 26.88 | 1 / 15 | | 1×20 + 31×1 | Aligned CB | 743.30 | 491.93 | 34.48 | 32 / 15 | | 1×20 + 31×1 | **Auto** | 90.62 | 394.78 | N/A | 1 / 15 | | 1×128 + 31×4 | No-CB | 490.08 | 399.13 | N/A | 1 / 15 | | 1×128 + 31×4 | Equal-width CB | 490.85 | 482.08 | 27.09 | 1 / 15 | | 1×128 + 31×4 | Aligned CB | 889.01 | 483.21 | 34.83 | 32 / 15 | | 1×128 + 31×4 | **Auto** | 63.80 | 485.00 | 31.43 | 2 / 15 | | 16, 32, …, 512 | No-CB | 1918.03 | 531.64 | N/A | 1 / 15 | | 16, 32, …, 512 | Equal-width CB | 1918.95 | 594.68 | 27.32 | 1 / 15 | | 16, 32, …, 512 | Aligned CB | 1648.16 | 593.25 | 35.91 | 32 / 15 | | 16, 32, …, 512 | **Auto** | 1301.59 | 593.31 | 32.49 | 6 / 15 | The 128/4 and 512/16 workloads reduce prefill Forwards from 32 on aligned CB to **2 on Auto**. Spread uses **6 buckets**. All CB paths in these performance runs have zero trims and zero moved bytes; ordinary generation has no static-cache statistics. This makes no new trimming-speedup claim. </details> ## Validation and limits A separate microbenchmark measures **planner overhead**, without model Prefill: for 128/256 requests of 65,536 tokens, CPU DP takes **0.48/1.82 ms** on average with the default per-bucket token limit. It still checks all 8,256/32,896 candidate segments; over-limit segments skip cost evaluation. For 32 requests, DP takes 0.089 ms at length512 and 0.040 ms at length65,536 (0.078 ms with no token limit). GPU mask validation/length extraction for 32×65,536 takes 0.83 ms separately. Measurements use ten repeated timing samples after warmup. **These are not 64k model Prefill timings or the complete Auto-path overhead.** **The original validation passed 105 related rollout/cache tests**, including independent exhaustive partition comparisons for DP, device-backed tiny-Qwen tests for grouped prefill and refill, both cache APIs, effective generation-config/legacy refresh, repetition history, EOS/PAD masks, capacity bounds and external-wall-time checks for profiling. Existing tests cover retirement, compaction and trimming. Independent mixed-bucket checks also exercised live-survivor refill and trimming on/off against padded per-request HF generation. One pre-existing native shared-prefill extension test is excluded because the host has no CUDA compiler; **all added tests ran**. Changed-file pre-commit and `git diff --check` pass. Repeated local review ended with **no actionable findings**. ```sh python -m pytest -q \ tests/unit/runtime/rollout/test_hybrid_engine_rollout.py \ tests/unit/runtime/rollout/test_continuous_batching.py \ tests/unit/runtime/rollout/test_rollout_interface.py \ tests/unit/utils/test_static_cache.py \ -k 'not native_repeat_kv_cache_fp16_reverse_copy' ``` The prefill token limit does **not** bound allocated decode KV memory. A single prompt over that limit raises an explicit error; there is no chunked prefill. **64k prompts require a model context limit covering prompt plus output budget, sufficient KV capacity and temporary memory.** The measured checkpoint is configured for 32k. Planner/position-limit boundaries are tested; real 64k inference performance is not claimed. Performance evidence uses unsharded HF models behind `HybridEngineRollout`; the additional calibrated and real-question checks are listed below. Other model/device configurations, FP16, native kernel injection, ZeRO/TP training integration and production arrival/EOS distributions remain unmeasured. Controlled performance runs do not exercise staggered EOS or trimming; those have correctness coverage. No new dependency, kernel, online learner or trimming redesign. Related foundation: [deepspeedai#8368](deepspeedai#8368); profiling: [deepspeedai#8494](deepspeedai#8494). ## Reviewer follow-up [Commit 65a9115](deepspeedai@65a9115) propagates the resolved generation configuration through continuous Prefill, Decode and refill, and omits `output_logits` from legacy manual generation. **27 targeted tests pass**, including six new cases covering both Auto routes, both cache APIs and manual equal-width/aligned compatibility. The new tests reproduced four failures before the fix. The compatibility test uses a strict generation wrapper that rejects the unsupported keyword. Calibration and evaluation tooling moved to [DeepSpeedExamples deepspeedai#1018](deepspeedai/DeepSpeedExamples#1018), with its fitting regression test and usage documentation. **Merge the Examples companion before this PR so the documentation link is available.** The Examples companion includes MHA-head fallback and validates a source fingerprint covering rollout, planning and static-cache code. A seeded tiny-model GPU smoke check completed calibration and all four saved-config sweep workloads with matching output IDs on every path. Two independent local reviews returned no findings. The calibrated sweep from the preceding routing-overhead fix is separate from the original 16-token experiments above. It uses **two EOS-disabled output tokens**, SDPA and one GPU, with independent uniform calibration shapes and one unchanged coefficient set per configuration: | Configuration | 128×16 | 1×128 + 31×4 | 16..512 | 1×512 + 31×16 | Worst regression | | --- | ---: | ---: | ---: | ---: | ---: | | Tiny Qwen2 / FP32 | 1.0220× | 0.9685× | 0.9539× | 0.9647× | +2.205% | | Qwen2.5-3B / FP32 | 1.0005× | 0.2345× | 0.7808× | 0.2265× | +0.054% | | Qwen3-32B / BF16 | 1.0027× | 0.3881× | 0.7189× | 0.1811× | +0.273% | Values are **Auto E2E / fastest manual baseline E2E**. The two larger models select no-CB for 128×16 and bucketed CB for the other three cases; the tiny model selects no-CB throughout. Auto matches ordinary output IDs in all twelve cases. Target-model means use three interleaved repetitions after warmup; the tiny test uses ten. [Full sweep details and calibration limits](deepspeedai#8786 (comment)). A separate [Qwen3-32B real-question check](deepspeedai#8786 (comment)) selected no-CB on both workloads: MATH-500 averaged **31.452 s / 31.393 s** for ordinary/Auto over three runs; ARC-AGI-2 measured **237.560 s / 237.888 s** in one paired run. All eight responses matched. These checks cover complete generation calls, excluding tokenization, loading, rewards and training. A real tiny-OPT calibration additionally exposed a missing HF cache capability flag. [Commit ecacbce](deepspeedai@ecacbce) adds `is_compileable=True` to the static-cache container, matching its existing static layers. **Six targeted GPU checks pass**, including new OPT manual/Auto comparisons with padded prompts and refill against the HF oracle, plus existing Qwen bucket/cache-path checks. The three Examples regression tests cover fitting, complete OPT calibration and saved-source compatibility. Both local reviews found no issues in this follow-up. ## Final correctness and performance acceptance [Commit d4def3d](deepspeedai@d4def3d) includes the final fixes: non-default repetition penalties are calculated in FP32 like HF, and unknown public generation switches are rejected before routing. The default-penalty path retains its early return. **Five focused BF16 checks and six unknown-setting CPU cases pass.** The Examples CLI now requires complete metadata for saved calibration files. Both independent correctness and performance reviews returned no findings after these fixes. A new calibration and sweep were run against measured commit `d4def3d`. **Qwen3-32B, BF16, SDPA, one GPU; two EOS-disabled output tokens; one warmup plus three interleaved timed runs per path.** Calibration and held-out evaluation use separate shapes. The calibration file, sweep and that measured revision's rollout/planner/static-cache sources have the same combined SHA256: `f9d79a43d2e6c0e3fef164ef11e81a26c643820beb1a0d3f1a86f41a46dbdd0c`. | Workload | No-CB ms | Equal-width CB ms | Aligned CB ms | Auto ms | Auto / best | Auto path | | --- | ---: | ---: | ---: | ---: | ---: | --- | | 128×16 | **269.83** | 439.91 | 5717.89 | 270.03 | 1.000743× | No-CB | | 1×128 + 31×4 | **496.15** | 540.76 | 1501.57 | **191.96** | 0.386886× | 2 buckets | | 16,32,…,512 | 1962.87 | 2009.18 | **1864.55** | **1336.57** | 0.716828× | 6 buckets | | 1×512 + 31×16 | 1962.56 | 2005.33 | **1575.08** | **288.31** | 0.183045× | 2 buckets | **All four pass the 5% acceptance tolerance; worst measured regression is +0.07428%. Auto output IDs match ordinary generation in every case**, and both routes are exercised. Means and ratios were independently recomputed from raw samples. The legacy aligned spread baseline retains its previously recorded second-token difference on one row; Auto matches no-CB there. Fixed output length prevents this numerical-sensitive baseline difference from changing the amount of timed work. The fitted coefficients are fixed=28.791679 ms, token=0.093362064 ms, attention=0.010090454 ms/GFLOP, KV=0.056288540 ms/MiB and additional decode=17.144541 ms/step. The earlier tables remain historical measurements on their original versions. This final check does not establish long-context, natural-EOS, 256-request E2E or dynamic-arrival performance. Auto remains disabled by default. [Commit a0588b7](deepspeedai@a0588b7) then fixes manual aligned prefill to retain the original padded repetition history. Its focused GPU regression reproduced the mismatch and now passes. Auto already passed this same mapping, and the measured default-penalty paths do not read it; independent correctness and performance delta reviews returned no findings. The table remains attributed to `d4def3d`, with no timings or hashes relabeled as a new-HEAD measurement. The Examples benchmark explicitly requires Transformers >=4.56; the library's existing compatibility scope is unchanged. ## Review invitation cc @nathon-lee @delock — this follows your continuous-batching, profiling, and aligned-cache work in deepspeedai#8368, deepspeedai#8494, and deepspeedai#8497. Could you review the automatic route selection, bucket/refill cache semantics, and cost defaults? ## November refresh Rebased onto `b62afa8` and squashed to one commit; the PR is now mergeable. The refresh keeps deepspeedai#8732 graph state, deepspeedai#8839 last-position prefill logits, deepspeedai#8933 accelerator-neutral graph calls and deepspeedai#8852 `DeepSpeedStaticCache.get_query_offset()`. deepspeedai#8980's explicit accelerator opt-in is respected: a backend without `is_graph_capture_supported()` falls back to eager before any routing decision. *Superseded by 8a9641b:* Auto now combines eager bucketed prefill with fixed-slot graph decode (see the November 11 update). SegKI (deepspeedai#8602) is not part of this PR. Validation, 8×H200-class node with one GPU in use (NVIDIA H20Z, PyTorch 2.9.1+cu128, CUDA 12.8, Transformers 4.57.1): | Check | Result | | --- | --- | | rollout/cache suite | 159 passed, 3 skipped, 1 deselected | | Skips | Three Transformers 5.20-only query-offset cases added by deepspeedai#8852 | | Deselected | Native shared-prefill kernel test; the test host has no CUDA compiler | | Changed-file pre-commit | Passed | Graph replay, resolved-config propagation on both routes (FP32/BF16), DP bucket/refill oracles, Auto/Graph rejection (since replaced by joint routing) and manual graph dispatch all pass. No new performance numbers are claimed: the tables above were measured before deepspeedai#8839/deepspeedai#8732, so refreshed baselines need a separate calibration run if wanted. AI assistance was used for the rebase, conflict resolution, test execution and drafting of this description; the submitting human reviewed the diff and the results above. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: SuperGoodGame <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
Adds adaptive prefill calibration and evaluation tooling alongside the existing rollout benchmark in
benchmarks/opsd. Requires Transformers >=4.56 and DeepSpeed PR #8786; the README explains configuration loading, the short-context calibration range and deployment validation.Calibration derives ordinary MHA KV heads from query heads when the dedicated field is absent. The CLI requires a complete metadata object for supplied calibration files; cost-only artifacts are rejected, while applications can still pass hand-tuned coefficients directly to the rollout configuration. Saved configurations are checked against the model/device/library metadata and a combined source hash of rollout, bucket planning and static-cache implementations; stale or missing hashes require recalibration.
Validation: three regression tests pass, covering non-negative fitting, real tiny-OPT calibration on a GPU, and CLI acceptance/rejection of matching, changed, missing and cache-only-changed source hashes. Six targeted GPU oracle checks in the DeepSpeed companion validate OPT/Qwen generation through the static cache. Python lint and diff checks pass. The original seeded tiny-Qwen2 calibration/sweep smoke check also completed all four workloads with matching output IDs on every path.