Repository navigation
[feat] Introduce Rust Controller backend - #223
0oshowero0 wants to merge 35 commits into
Conversation
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 162693eb88
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
ad85869 to
af80186
Compare
…ariants (#224) ## Motivation While building a formal specification of `TransferQueueController` to verify metadata parity for #220 / #223, several edge-case invariant bugs surfaced in the Python Controller across `force_fetch` readiness projection, slot recycling & pre-allocation ownership, nested field schema compaction, and duplicate KV key handling. This PR fixes these reachable invariants directly on `main` with minimal targeted guards. ## Changes - **`force_fetch` readiness with unregistered fields (`TransferQueueController.generate_batch_meta`)**: - Require all requested `data_fields` to be registered in `partition.field_name_mapping` before computing `production_status` in `mode="force_fetch"` (also used by `link`). Previously, unregistered fields were filtered out of `field_indices`, causing `production_status` to evaluate to `1` (`BatchMeta.is_ready == True`) when at least one other requested field was ready, and making readiness depend on whether a missing field had never been written (`1`) vs cleared via `kv_clear_fields` (`0`). - **Partition slot ownership & pre-allocated slot promotion (`update_production_status` & `set_custom_meta`)**: - Validate `global_indexes` against active and pre-allocated partition slots (`partition.global_indexes` and `partition.pre_allocated_global_indexes`, in $O(\text{batch size})$ without copying partition sets) in `TransferQueueController.update_production_status`, rejecting recycled or foreign-partition `GlobalIndex` updates while still permitting `force_fetch` writes onto pre-allocated slots. - Filter `TransferQueueController.set_custom_meta` to active or pre-allocated slots of the target partition so late `set_custom_meta` calls on cleared slots cannot leak ghost tags onto newly inserted samples when `GlobalIndexManager` reuses the slot. - Remove written indices from `pre_allocated_global_indexes` when `DataPartitionStatus.update_production_status` activates them into `global_indexes`, preventing subsequent `mode="insert"` allocations from handing out already-populated pre-allocated slots again. - **Nested field schema collapse guard (`FieldMeta.remove_samples`)**: - Collapse `is_nested` back to `False` only when `not self.is_non_tensor`, `len(self.per_sample_shapes) == len(self.global_indexes)`, and `None not in remaining_shapes`, preventing premature collapse into a uniform tensor shape while non-tensor or `None`-shape samples remain in the field. - **Duplicate KV keys (`kv_retrieve_meta` & `kv_link`)**: - Deduplicate missing keys in `TransferQueueController.kv_retrieve_meta(create=True)` so repeated keys in a single call do not allocate and leak unreachable orphan `GlobalIndex` slots. - Reject duplicate `keys` in `TransferQueueController.kv_link` (matching `TransferQueueController.link` and `tq.kv_link`), preventing `async_kv_link` calls with repeated keys from leaking orphan links in `linked_by` that permanently pin `PUBLIC_PARTITION` rows. ## Validation - `ruff check`, `ruff format --check`, `mypy`, and `python -m compileall -q transfer_queue tutorial tests` passed. - Added unit and Ray controller regression tests (`TestMetadataInvariants` in `tests/test_controller_data_partitions.py` and `test_controller_metadata_and_slot_ownership_invariants` in `tests/test_controller.py`), covering unregistered `force_fetch`/`link` fields, recycled slot rejection, pre-allocated slot `set_custom_meta` & promotion under `TQ_PRE_ALLOC_SAMPLE_NUM > 1`, mixed nested/non-tensor sample removal, and duplicate KV keys. Signed-off-by: 0oshowero0 <[email protected]> Co-authored-by: Cursor <[email protected]>
Encode BatchMeta with MessagePack extension 6 and encode standalone field schemas with native values. Carry per-row tags and per-field backend metadata as codec-tagged blobs. Cover the wire format and round trips over real ZMQ. Client and Controller use the same package version. Signed-off-by: 0oshowero0 <[email protected]>
Restore uniform and per-sample shapes when constructing Python BatchMeta from MessagePack. Keep scalar shapes and missing shapes covered by the round-trip tests. Signed-off-by: 0oshowero0 <[email protected]>
Signed-off-by: 0oshowero0 <[email protected]>
Send canonical dtype names in metadata RPCs and restore Torch dtypes at Python entry points. Resolve known legacy dtype names when loading Controller checkpoints and keep protocol tests focused. Signed-off-by: 0oshowero0 <[email protected]>
Encode standalone tags and backend metadata as typed extensions so the shared codec can restore them. Build SET_CUSTOM_META from one tag snapshot grouped by partition. Signed-off-by: 0oshowero0 <[email protected]>
Use the existing codec and payload pair at metadata RPC boundaries to drop the extra MessagePack extension layer. Signed-off-by: 0oshowero0 <[email protected]>
Warn once when restoring legacy dtype names and tell callers to re-save with the current release. Signed-off-by: 0oshowero0 <[email protected]>
Share field schema encoding and decoding across metadata RPCs. Restore tuple shapes before recording data and when loading older Controller checkpoints. Signed-off-by: 0oshowero0 <[email protected]>
Move nested accelerator tensors to CPU during metadata pickling. Apply the same rule to BatchMeta when a message falls back to pickle. Signed-off-by: 0oshowero0 <[email protected]>
Include the frame summary when a message omits a required envelope field. Signed-off-by: 0oshowero0 <[email protected]>
Accept scalar and flat-list tag values. Validate tags before writing or restoring state and pass empty metadata values through natively. Signed-off-by: 0oshowero0 <[email protected]>
Normalize BatchMeta indexes at the wire boundary so custom samplers do not trigger whole-message pickle fallback. Signed-off-by: 0oshowero0 <[email protected]>
Reuse schema dtype parsing during checkpoint preflight so an invalid legacy dtype cannot replace Controller state or restore storage payloads. Signed-off-by: 0oshowero0 <[email protected]>
Send backend values directly, normalize indexes before transport, and validate row tags before they can enter the wire codec or storage. Keep BatchMeta.extra_info in Python and Ray serialization. Signed-off-by: 0oshowero0 <[email protected]>
Prepare the Controller snapshot once before restoring storage. Apply that snapshot on success or release it on failure. Signed-off-by: 0oshowero0 <[email protected]>
Remove custom tag/backend validation types and helpers, restore original client/interface and single-step checkpoint restore flows, and fail at runtime when BatchMeta cannot be encoded as native MessagePack. Signed-off-by: 0oshowero0 <[email protected]> Co-authored-by: Cursor <[email protected]>
… deprecations Keep per_sample_shapes list-to-dict indexing inside StorageManager.notify_data_update, clarify the legacy checkpoint FutureWarning, and tag all FutureWarning removals with TODO(0.2.0). Signed-off-by: 0oshowero0 <[email protected]> Co-authored-by: Cursor <[email protected]>
…estores Reject non-native per-sample tags (e.g. tensors, NumPy scalars, >64-bit ints) with ValueError before registering keys or writing payloads, and check custom_meta during legacy Controller checkpoint preflight so accepted writes never leave unreadable rows. Co-authored-by: Cursor <[email protected]> Signed-off-by: 0oshowero0 <[email protected]> Co-authored-by: Cursor <[email protected]>
Signed-off-by: 0oshowero0 <[email protected]> Co-authored-by: Cursor <[email protected]>
Signed-off-by: 0oshowero0 <[email protected]>
Signed-off-by: 0oshowero0 <[email protected]>
Signed-off-by: 0oshowero0 <[email protected]>
Signed-off-by: 0oshowero0 <[email protected]>
Signed-off-by: 0oshowero0 <[email protected]>
…s wire format - Collapse the 4-thread mpsc/inproc ZMQ pipeline into a single adaptive spin-then-poll worker thread that survives transient send/decode errors. - Replace in-process loopback TCP ZMQ requests on RustControllerActor with direct PyO3 methods on ControllerEngine, removing all 7 private ZMQRequestType enum values. - Unify GET_PRODUCTION and GET_CONSUMPTION wire replies across Python and Rust controllers using native index lists and flat status bytes, removing production_status_columns and client-side backend branching. Signed-off-by: 0oshowero0 <[email protected]> Co-authored-by: Cursor <[email protected]>
Signed-off-by: 0oshowero0 <[email protected]>
Signed-off-by: 0oshowero0 <[email protected]>
Signed-off-by: 0oshowero0 <[email protected]> Co-authored-by: Cursor <[email protected]>
- Add rust-toolchain.toml (1.100.0) and make the Rust extension non-optional. - Support TQ_CONTROLLER_BACKEND via config interpolation and remove the sed-based prepare-controller CI action. - Run MooncakeStore and Yuanrong E2E suites with both Python and Rust controllers. - Add cargo fmt, check_license, and check_docstrings hooks to pre-commit and drop heavy package installation in pre-commit CI. - Extend check_license.py and sanity workflow paths to cover crates/**/*.rs. - Switch Rust CI unit test step to cargo test -p tq-controller --locked and add pure Rust state unit tests. Co-authored-by: Cursor <[email protected]> Signed-off-by: 0oshowero0 <[email protected]>
Upload wheels onto an existing release when publishing from the GitHub Releases UI, and allow manual workflow_dispatch runs on tags to publish. Co-authored-by: Cursor <[email protected]> Signed-off-by: 0oshowero0 <[email protected]>
Co-authored-by: Cursor <[email protected]> Signed-off-by: 0oshowero0 <[email protected]>
Signed-off-by: 0oshowero0 <[email protected]> Co-authored-by: Cursor <[email protected]>
Signed-off-by: 0oshowero0 <[email protected]> Co-authored-by: Cursor <[email protected]>
…r parity tests Signed-off-by: 0oshowero0 <[email protected]> Co-authored-by: Cursor <[email protected]>
…ariants Signed-off-by: 0oshowero0 <[email protected]> Co-authored-by: Cursor <[email protected]>
25f9877 to
93ba018
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 93ba0188ca
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if rust: | ||
| ray.get(_TQ_CONTROLLER.validate_checkpoint.remote(str(controller_path))) | ||
| client.load_storage_checkpoint(str(checkpoint_dir)) |
There was a problem hiding this comment.
Validate Python checkpoints before replacing storage
When restoring a pre-upgrade Python/SimpleStorage checkpoint whose tags contain a tensor or 2**70 (previously accepted), this branch skips prevalidation and replaces storage before TransferQueueController.load_checkpoint() rejects the tags. That exception occurs before the saved partitions and index allocator are installed, leaving the current metadata pointing at unrelated checkpoint payloads even though tq.load_checkpoint() failed. Validate Python controller checkpoints before client.load_storage_checkpoint() as well, or safely migrate those tags before mutating storage.
Useful? React with 👍 / 👎.
Motivation
Following the native MessagePack metadata wire format in #221, this PR implements the Rust Controller engine proposed in #220 behind
controller.backend="rust"("python"remains the default).Under non-blocking consumer polling, high-frequency
GET_META,NOTIFY_DATA_UPDATE, andKV_*RPCs converge on a single GIL-bound Python thread inTransferQueueControllerand scan sparsetorch.int8status tensors on every poll. Embedding a native Rust metadata engine in the same Ray actor process removes GIL contention and tensor mask allocations on the ZMQ hot path while keeping Ray actor discovery, storage backends, and Prometheus/Grafana observability unchanged.Stacked on #221.
Changes
crates/controller,crates/controller-py,transfer_queue/rust_controller.py):transfer_queue._controller_engineas a non-optionalcp310-abi3PyO3 extension and host it insideRustControllerActorunder the sameTransferQueueControllerRay actor name.handshake_socketandrequest_handle_socketZMQROUTERsockets in a native Rust adaptive spin-then-poll worker loop without acquiring the GIL, and expose actor-side lifecycle/inspection methods (list_partitions,get_partition_snapshot,save_checkpoint,load_checkpoint,validate_checkpoint,shutdown) directly via PyO3.u64bitmap state & RPC parity (SequentialSampler):IndexAllocator(per-storage-unit reusable FIFO queues) andPartitionStateover dense row slots andu64bitmaps for field readiness and per-task consumption.HANDSHAKE,GET_METAininsert/fetch/force_fetch,NOTIFY_DATA_UPDATE,SET_CUSTOM_META,CLEAR_META,CLEAR_PARTITION,GET_PRODUCTION,GET_CONSUMPTION,RESET_CONSUMPTION,GET_LIST_PARTITIONS,GET_PARTITION_META), KV RPCs (KV_RETRIEVE_META,KV_RETRIEVE_KEYS,KV_LIST,KV_EXISTS,KV_DESCRIBE,KV_PREPARE_UPDATE,KV_CLEAR_FIELDS,VALIDATE_DUMP_SCHEMA), shared-field link RPCs (LINK,KV_LINK,CHECK_PUBLIC_WRITE,MARK_UNLINKED_PUBLIC_CLEARING), and atomic request/partition metrics snapshots exported throughTQMetricsExporter.GET_PRODUCTIONandGET_CONSUMPTIONreplies across Python and Rust controllers to transport nativeglobal_indexinteger lists and compact row-major statusbytesinstead oftorch.Tensor.controller_state.msgpack), validating snapshots before restoring storage and rejecting cross-backend checkpoint loads (controller_state.pklvscontroller_state.msgpack) withValueError.1.100.0(minimalprofile) inrust-toolchain.toml, build and verifycp310-abi3manylinux_2_28wheels on Linuxx86_64andaarch64(wheel-package.yml), and wire tag pushes into GitHub Releases (release.yml).cargo fmt, license header, and docstring checks to.pre-commit-config.yamlandrust-controller-ci.yml; parallelize unit andpython/rustE2E jobs inrun-tests.yml, and cover both controller backends in tutorials, recipes, andperftest.Scope & Current Limitations
controller.backenddefaults to"python". Pass{"controller": {"backend": "rust"}}totq.init(conf)to enable the Rust Controller.SequentialSampler(requesting other samplers raisesValueError); additional built-in samplers and custom PythonBaseSamplercallbacks from [RFC] A Concurrent Rust Request Engine for TransferQueueController #220 will follow in subsequent PRs.controller_state.pklfor Python vscontroller_state.msgpackfor Rust).Validation
cargo fmt --all --check,cargo clippy --workspace --all-targets --locked -- -D warnings, andcargo test -p tq-controller --lockedpassed.ruff,mypy,compileall, docstring, and Apache-2.0 license checks passed across Python and Rust sources.cp310-abi3manylinux_2_28wheels on Linuxx86_64andaarch64.tests/test_rust_controller.py), tutorials, recipes,perftest, and E2E suites acrossSimpleStorage,MooncakeStore, andYuanrongwith bothpythonandrustcontrollers.