Skip to content

[feat] Introduce Rust Controller backend - #223

Open
0oshowero0 wants to merge 35 commits into
Ascend:mainfrom
0oshowero0:feat/rust-controller
Open

0oshowero0 wants to merge 35 commits into
Ascend:mainfrom
0oshowero0:feat/rust-controller

Conversation

@0oshowero0

@0oshowero0 0oshowero0 commented Nov 22, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Following the native MessagePack metadata wire format in #221, this PR implements the Rust Controller engine proposed in #220 behind controller.backend="rust" ("python" remains the default).

Under non-blocking consumer polling, high-frequency GET_META, NOTIFY_DATA_UPDATE, and KV_* RPCs converge on a single GIL-bound Python thread in TransferQueueController and scan sparse torch.int8 status tensors on every poll. Embedding a native Rust metadata engine in the same Ray actor process removes GIL contention and tensor mask allocations on the ZMQ hot path while keeping Ray actor discovery, storage backends, and Prometheus/Grafana observability unchanged.

Stacked on #221.

Changes

  • In-process Rust metadata engine & PyO3 extension (crates/controller, crates/controller-py, transfer_queue/rust_controller.py):
    • Build transfer_queue._controller_engine as a non-optional cp310-abi3 PyO3 extension and host it inside RustControllerActor under the same TransferQueueController Ray actor name.
    • Run handshake_socket and request_handle_socket ZMQ ROUTER sockets in a native Rust adaptive spin-then-poll worker loop without acquiring the GIL, and expose actor-side lifecycle/inspection methods (list_partitions, get_partition_snapshot, save_checkpoint, load_checkpoint, validate_checkpoint, shutdown) directly via PyO3.
  • Compact u64 bitmap state & RPC parity (SequentialSampler):
    • Implement IndexAllocator (per-storage-unit reusable FIFO queues) and PartitionState over dense row slots and u64 bitmaps for field readiness and per-task consumption.
    • Support batch lifecycle RPCs (HANDSHAKE, GET_META in insert/fetch/force_fetch, NOTIFY_DATA_UPDATE, SET_CUSTOM_META, CLEAR_META, CLEAR_PARTITION, GET_PRODUCTION, GET_CONSUMPTION, RESET_CONSUMPTION, GET_LIST_PARTITIONS, GET_PARTITION_META), KV RPCs (KV_RETRIEVE_META, KV_RETRIEVE_KEYS, KV_LIST, KV_EXISTS, KV_DESCRIBE, KV_PREPARE_UPDATE, KV_CLEAR_FIELDS, VALIDATE_DUMP_SCHEMA), shared-field link RPCs (LINK, KV_LINK, CHECK_PUBLIC_WRITE, MARK_UNLINKED_PUBLIC_CLEARING), and atomic request/partition metrics snapshots exported through TQMetricsExporter.
  • Unified status wire format & native Rust checkpoints:
    • Unify GET_PRODUCTION and GET_CONSUMPTION replies across Python and Rust controllers to transport native global_index integer lists and compact row-major status bytes instead of torch.Tensor.
    • Persist Rust Controller checkpoints as versioned native MessagePack (controller_state.msgpack), validating snapshots before restoring storage and rejecting cross-backend checkpoint loads (controller_state.pkl vs controller_state.msgpack) with ValueError.
  • Packaging, toolchain & CI:
    • Pin Rust 1.100.0 (minimal profile) in rust-toolchain.toml, build and verify cp310-abi3 manylinux_2_28 wheels on Linux x86_64 and aarch64 (wheel-package.yml), and wire tag pushes into GitHub Releases (release.yml).
    • Add cargo fmt, license header, and docstring checks to .pre-commit-config.yaml and rust-controller-ci.yml; parallelize unit and python/rust E2E jobs in run-tests.yml, and cover both controller backends in tutorials, recipes, and perftest.

Scope & Current Limitations

  • Opt-in configuration: controller.backend defaults to "python". Pass {"controller": {"backend": "rust"}} to tq.init(conf) to enable the Rust Controller.
  • Sampler support: The Rust Controller currently supports SequentialSampler (requesting other samplers raises ValueError); additional built-in samplers and custom Python BaseSampler callbacks from [RFC] A Concurrent Rust Request Engine for TransferQueueController #220 will follow in subsequent PRs.
  • Checkpoint format: Controller checkpoints are backend-specific (controller_state.pkl for Python vs controller_state.msgpack for Rust).

Validation

  • cargo fmt --all --check, cargo clippy --workspace --all-targets --locked -- -D warnings, and cargo test -p tq-controller --locked passed.
  • ruff, mypy, compileall, docstring, and Apache-2.0 license checks passed across Python and Rust sources.
  • Built and tested cp310-abi3 manylinux_2_28 wheels on Linux x86_64 and aarch64.
  • Passed unit tests, Python/Rust parity and boundary tests (tests/test_rust_controller.py), tutorials, recipes, perftest, and E2E suites across SimpleStorage, MooncakeStore, and Yuanrong with both python and rust controllers.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 22, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-22T15:36:42.021963Z 93ba018 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 162693eb88

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread transfer_queue/client.py Resolved
@0oshowero0
0oshowero0 force-pushed the feat/rust-controller branch 2 times, most recently from ad85869 to af80186 Compare November 22, 2026 10:29
@0oshowero0 0oshowero0 changed the title [WIP][refactor] Introduce Rust controller [feat] Introduce Rust Controller backend Nov 22, 2026
0oshowero0 added a commit that referenced this pull request Nov 22, 2026
…ariants (#224)

## Motivation

While building a formal specification of `TransferQueueController` to
verify metadata parity for #220 / #223, several edge-case invariant bugs
surfaced in the Python Controller across `force_fetch` readiness
projection, slot recycling & pre-allocation ownership, nested field
schema compaction, and duplicate KV key handling. This PR fixes these
reachable invariants directly on `main` with minimal targeted guards.

## Changes

- **`force_fetch` readiness with unregistered fields
(`TransferQueueController.generate_batch_meta`)**:
- Require all requested `data_fields` to be registered in
`partition.field_name_mapping` before computing `production_status` in
`mode="force_fetch"` (also used by `link`). Previously, unregistered
fields were filtered out of `field_indices`, causing `production_status`
to evaluate to `1` (`BatchMeta.is_ready == True`) when at least one
other requested field was ready, and making readiness depend on whether
a missing field had never been written (`1`) vs cleared via
`kv_clear_fields` (`0`).
- **Partition slot ownership & pre-allocated slot promotion
(`update_production_status` & `set_custom_meta`)**:
- Validate `global_indexes` against active and pre-allocated partition
slots (`partition.global_indexes` and
`partition.pre_allocated_global_indexes`, in $O(\text{batch size})$
without copying partition sets) in
`TransferQueueController.update_production_status`, rejecting recycled
or foreign-partition `GlobalIndex` updates while still permitting
`force_fetch` writes onto pre-allocated slots.
- Filter `TransferQueueController.set_custom_meta` to active or
pre-allocated slots of the target partition so late `set_custom_meta`
calls on cleared slots cannot leak ghost tags onto newly inserted
samples when `GlobalIndexManager` reuses the slot.
- Remove written indices from `pre_allocated_global_indexes` when
`DataPartitionStatus.update_production_status` activates them into
`global_indexes`, preventing subsequent `mode="insert"` allocations from
handing out already-populated pre-allocated slots again.
- **Nested field schema collapse guard (`FieldMeta.remove_samples`)**:
- Collapse `is_nested` back to `False` only when `not
self.is_non_tensor`, `len(self.per_sample_shapes) ==
len(self.global_indexes)`, and `None not in remaining_shapes`,
preventing premature collapse into a uniform tensor shape while
non-tensor or `None`-shape samples remain in the field.
- **Duplicate KV keys (`kv_retrieve_meta` & `kv_link`)**:
- Deduplicate missing keys in
`TransferQueueController.kv_retrieve_meta(create=True)` so repeated keys
in a single call do not allocate and leak unreachable orphan
`GlobalIndex` slots.
- Reject duplicate `keys` in `TransferQueueController.kv_link` (matching
`TransferQueueController.link` and `tq.kv_link`), preventing
`async_kv_link` calls with repeated keys from leaking orphan links in
`linked_by` that permanently pin `PUBLIC_PARTITION` rows.

## Validation

- `ruff check`, `ruff format --check`, `mypy`, and `python -m compileall
-q transfer_queue tutorial tests` passed.
- Added unit and Ray controller regression tests
(`TestMetadataInvariants` in `tests/test_controller_data_partitions.py`
and `test_controller_metadata_and_slot_ownership_invariants` in
`tests/test_controller.py`), covering unregistered `force_fetch`/`link`
fields, recycled slot rejection, pre-allocated slot `set_custom_meta` &
promotion under `TQ_PRE_ALLOC_SAMPLE_NUM > 1`, mixed nested/non-tensor
sample removal, and duplicate KV keys.

Signed-off-by: 0oshowero0 <[email protected]>
Co-authored-by: Cursor <[email protected]>
0oshowero0 and others added 23 commits November 22, 2026 22:43
Encode BatchMeta with MessagePack extension 6 and encode standalone field schemas with native values. Carry per-row tags and per-field backend metadata as codec-tagged blobs.

Cover the wire format and round trips over real ZMQ. Client and Controller use the same package version.

Signed-off-by: 0oshowero0 <[email protected]>
Restore uniform and per-sample shapes when constructing Python BatchMeta from MessagePack. Keep scalar shapes and missing shapes covered by the round-trip tests.

Signed-off-by: 0oshowero0 <[email protected]>
Send canonical dtype names in metadata RPCs and restore Torch dtypes at Python entry points. Resolve known legacy dtype names when loading Controller checkpoints and keep protocol tests focused.

Signed-off-by: 0oshowero0 <[email protected]>
Encode standalone tags and backend metadata as typed extensions so the shared codec can restore them. Build SET_CUSTOM_META from one tag snapshot grouped by partition.

Signed-off-by: 0oshowero0 <[email protected]>
Use the existing codec and payload pair at metadata RPC boundaries to drop the extra MessagePack extension layer.

Signed-off-by: 0oshowero0 <[email protected]>
Warn once when restoring legacy dtype names and tell callers to re-save with the current release.

Signed-off-by: 0oshowero0 <[email protected]>
Share field schema encoding and decoding across metadata RPCs. Restore tuple shapes before recording data and when loading older Controller checkpoints.

Signed-off-by: 0oshowero0 <[email protected]>
Move nested accelerator tensors to CPU during metadata pickling. Apply the same rule to BatchMeta when a message falls back to pickle.

Signed-off-by: 0oshowero0 <[email protected]>
Include the frame summary when a message omits a required envelope field.

Signed-off-by: 0oshowero0 <[email protected]>
Accept scalar and flat-list tag values. Validate tags before writing or restoring state and pass empty metadata values through natively.

Signed-off-by: 0oshowero0 <[email protected]>
Normalize BatchMeta indexes at the wire boundary so custom samplers do not trigger whole-message pickle fallback.

Signed-off-by: 0oshowero0 <[email protected]>
Reuse schema dtype parsing during checkpoint preflight so an invalid legacy dtype cannot replace Controller state or restore storage payloads.

Signed-off-by: 0oshowero0 <[email protected]>
Send backend values directly, normalize indexes before transport, and validate row tags before they can enter the wire codec or storage. Keep BatchMeta.extra_info in Python and Ray serialization.

Signed-off-by: 0oshowero0 <[email protected]>
Prepare the Controller snapshot once before restoring storage. Apply that snapshot on success or release it on failure.

Signed-off-by: 0oshowero0 <[email protected]>
Remove custom tag/backend validation types and helpers, restore original client/interface and single-step checkpoint restore flows, and fail at runtime when BatchMeta cannot be encoded as native MessagePack.

Signed-off-by: 0oshowero0 <[email protected]>
Co-authored-by: Cursor <[email protected]>
… deprecations

Keep per_sample_shapes list-to-dict indexing inside StorageManager.notify_data_update, clarify the legacy checkpoint FutureWarning, and tag all FutureWarning removals with TODO(0.2.0).

Signed-off-by: 0oshowero0 <[email protected]>
Co-authored-by: Cursor <[email protected]>
…estores

Reject non-native per-sample tags (e.g. tensors, NumPy scalars, >64-bit ints) with ValueError before registering keys or writing payloads, and check custom_meta during legacy Controller checkpoint preflight so accepted writes never leave unreadable rows.

Co-authored-by: Cursor <[email protected]>
Signed-off-by: 0oshowero0 <[email protected]>
Co-authored-by: Cursor <[email protected]>
0oshowero0 and others added 12 commits November 22, 2026 22:44
…s wire format

- Collapse the 4-thread mpsc/inproc ZMQ pipeline into a single adaptive spin-then-poll worker thread that survives transient send/decode errors.
- Replace in-process loopback TCP ZMQ requests on RustControllerActor with direct PyO3 methods on ControllerEngine, removing all 7 private ZMQRequestType enum values.
- Unify GET_PRODUCTION and GET_CONSUMPTION wire replies across Python and Rust controllers using native index lists and flat status bytes, removing production_status_columns and client-side backend branching.

Signed-off-by: 0oshowero0 <[email protected]>
Co-authored-by: Cursor <[email protected]>
- Add rust-toolchain.toml (1.100.0) and make the Rust extension non-optional.
- Support TQ_CONTROLLER_BACKEND via config interpolation and remove the sed-based prepare-controller CI action.
- Run MooncakeStore and Yuanrong E2E suites with both Python and Rust controllers.
- Add cargo fmt, check_license, and check_docstrings hooks to pre-commit and drop heavy package installation in pre-commit CI.
- Extend check_license.py and sanity workflow paths to cover crates/**/*.rs.
- Switch Rust CI unit test step to cargo test -p tq-controller --locked and add pure Rust state unit tests.

Co-authored-by: Cursor <[email protected]>
Signed-off-by: 0oshowero0 <[email protected]>
Upload wheels onto an existing release when publishing from the GitHub Releases UI, and allow manual workflow_dispatch runs on tags to publish.

Co-authored-by: Cursor <[email protected]>
Signed-off-by: 0oshowero0 <[email protected]>
…r parity tests

Signed-off-by: 0oshowero0 <[email protected]>
Co-authored-by: Cursor <[email protected]>
@0oshowero0

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 93ba0188ca

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +1618 to 1620
if rust:
ray.get(_TQ_CONTROLLER.validate_checkpoint.remote(str(controller_path)))
client.load_storage_checkpoint(str(checkpoint_dir))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Validate Python checkpoints before replacing storage

When restoring a pre-upgrade Python/SimpleStorage checkpoint whose tags contain a tensor or 2**70 (previously accepted), this branch skips prevalidation and replaces storage before TransferQueueController.load_checkpoint() rejects the tags. That exception occurs before the saved partitions and index allocator are installed, leaving the current metadata pointing at unrelated checkpoint payloads even though tq.load_checkpoint() failed. Validate Python controller checkpoints before client.load_storage_checkpoint() as well, or safely migrate those tags before mutating storage.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant