Repository navigation
[fix] Preserve Controller metadata, readiness, and slot ownership invariants - #224
Conversation
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
fc93309 to
3dfb288
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fc933099fa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| unallocated = [idx for idx in global_indexes if idx not in partition.global_indexes] | ||
| if unallocated: | ||
| logger.error(f"Unallocated global indexes {unallocated} for partition {partition_id}") | ||
| return False |
There was a problem hiding this comment.
Include reserved rows when checking partition ownership
When TQ_PRE_ALLOC_SAMPLE_NUM exceeds the number of inserted samples, get_metadata(..., mode="force_fetch") returns both active rows and still-reserved rows because it uses index_manager.get_indexes_for_partition. Passing that metadata to client.put now writes the payload but then fails the controller notification: reserved rows are absent from partition.global_indexes even though they are allocated to this partition. This rejects the entire batch, including its active rows. Check ownership against active plus pre-allocated indexes rather than treating reserved rows as recycled/unallocated; the new set_custom_meta filter also silently drops metadata for those reserved rows.
Useful? React with 👍 / 👎.
3dfb288 to
ee33a15
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ee33a154d9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| allocated = partition.global_indexes | partition.pre_allocated_global_indexes | ||
| unallocated = [idx for idx in global_indexes if idx not in allocated] |
There was a problem hiding this comment.
Check ownership without copying the whole partition
For large or heavily preallocated partitions receiving small kv_put/kv_update batches, this union copies every active and reserved index on each write notification, even when only one row changes. That adds O(partition size) work and temporary memory to the controller's serial request path; ordinary puts repeat the copy in set_custom_meta. Check membership directly in the two existing sets in both locations so ownership validation scales with the requested batch rather than the entire partition.
Useful? React with 👍 / 👎.
ee33a15 to
349ef0f
Compare
…ariants Signed-off-by: 0oshowero0 <[email protected]> Co-authored-by: Cursor <[email protected]>
349ef0f to
3ae586a
Compare
Motivation
While building a formal specification of
TransferQueueControllerto verify metadata parity for #220 / #223, several edge-case invariant bugs surfaced in the Python Controller acrossforce_fetchreadiness projection, slot recycling & pre-allocation ownership, nested field schema compaction, and duplicate KV key handling. This PR fixes these reachable invariants directly onmainwith minimal targeted guards.Changes
force_fetchreadiness with unregistered fields (TransferQueueController.generate_batch_meta):data_fieldsto be registered inpartition.field_name_mappingbefore computingproduction_statusinmode="force_fetch"(also used bylink). Previously, unregistered fields were filtered out offield_indices, causingproduction_statusto evaluate to1(BatchMeta.is_ready == True) when at least one other requested field was ready, and making readiness depend on whether a missing field had never been written (1) vs cleared viakv_clear_fields(0).update_production_status&set_custom_meta):global_indexesagainst active and pre-allocated partition slots (partition.global_indexesandpartition.pre_allocated_global_indexes, inTransferQueueController.update_production_status, rejecting recycled or foreign-partitionGlobalIndexupdates while still permittingforce_fetchwrites onto pre-allocated slots.TransferQueueController.set_custom_metato active or pre-allocated slots of the target partition so lateset_custom_metacalls on cleared slots cannot leak ghost tags onto newly inserted samples whenGlobalIndexManagerreuses the slot.pre_allocated_global_indexeswhenDataPartitionStatus.update_production_statusactivates them intoglobal_indexes, preventing subsequentmode="insert"allocations from handing out already-populated pre-allocated slots again.FieldMeta.remove_samples):is_nestedback toFalseonly whennot self.is_non_tensor,len(self.per_sample_shapes) == len(self.global_indexes), andNone not in remaining_shapes, preventing premature collapse into a uniform tensor shape while non-tensor orNone-shape samples remain in the field.kv_retrieve_meta&kv_link):TransferQueueController.kv_retrieve_meta(create=True)so repeated keys in a single call do not allocate and leak unreachable orphanGlobalIndexslots.keysinTransferQueueController.kv_link(matchingTransferQueueController.linkandtq.kv_link), preventingasync_kv_linkcalls with repeated keys from leaking orphan links inlinked_bythat permanently pinPUBLIC_PARTITIONrows.Validation
ruff check,ruff format --check,mypy, andpython -m compileall -q transfer_queue tutorial testspassed.TestMetadataInvariantsintests/test_controller_data_partitions.pyandtest_controller_metadata_and_slot_ownership_invariantsintests/test_controller.py), covering unregisteredforce_fetch/linkfields, recycled slot rejection, pre-allocated slotset_custom_meta& promotion underTQ_PRE_ALLOC_SAMPLE_NUM > 1, mixed nested/non-tensor sample removal, and duplicate KV keys.