Skip to content

[fix] Preserve Controller metadata, readiness, and slot ownership invariants - #224

Merged
0oshowero0 merged 1 commit into
Ascend:mainfrom
0oshowero0:fix/controller-metadata-invariants
Nov 22, 2026
Merged

0oshowero0 merged 1 commit into
Ascend:mainfrom
0oshowero0:fix/controller-metadata-invariants

Conversation

@0oshowero0

@0oshowero0 0oshowero0 commented Nov 22, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

While building a formal specification of TransferQueueController to verify metadata parity for #220 / #223, several edge-case invariant bugs surfaced in the Python Controller across force_fetch readiness projection, slot recycling & pre-allocation ownership, nested field schema compaction, and duplicate KV key handling. This PR fixes these reachable invariants directly on main with minimal targeted guards.

Changes

  • force_fetch readiness with unregistered fields (TransferQueueController.generate_batch_meta):
    • Require all requested data_fields to be registered in partition.field_name_mapping before computing production_status in mode="force_fetch" (also used by link). Previously, unregistered fields were filtered out of field_indices, causing production_status to evaluate to 1 (BatchMeta.is_ready == True) when at least one other requested field was ready, and making readiness depend on whether a missing field had never been written (1) vs cleared via kv_clear_fields (0).
  • Partition slot ownership & pre-allocated slot promotion (update_production_status & set_custom_meta):
    • Validate global_indexes against active and pre-allocated partition slots (partition.global_indexes and partition.pre_allocated_global_indexes, in $O(\text{batch size})$ without copying partition sets) in TransferQueueController.update_production_status, rejecting recycled or foreign-partition GlobalIndex updates while still permitting force_fetch writes onto pre-allocated slots.
    • Filter TransferQueueController.set_custom_meta to active or pre-allocated slots of the target partition so late set_custom_meta calls on cleared slots cannot leak ghost tags onto newly inserted samples when GlobalIndexManager reuses the slot.
    • Remove written indices from pre_allocated_global_indexes when DataPartitionStatus.update_production_status activates them into global_indexes, preventing subsequent mode="insert" allocations from handing out already-populated pre-allocated slots again.
  • Nested field schema collapse guard (FieldMeta.remove_samples):
    • Collapse is_nested back to False only when not self.is_non_tensor, len(self.per_sample_shapes) == len(self.global_indexes), and None not in remaining_shapes, preventing premature collapse into a uniform tensor shape while non-tensor or None-shape samples remain in the field.
  • Duplicate KV keys (kv_retrieve_meta & kv_link):
    • Deduplicate missing keys in TransferQueueController.kv_retrieve_meta(create=True) so repeated keys in a single call do not allocate and leak unreachable orphan GlobalIndex slots.
    • Reject duplicate keys in TransferQueueController.kv_link (matching TransferQueueController.link and tq.kv_link), preventing async_kv_link calls with repeated keys from leaking orphan links in linked_by that permanently pin PUBLIC_PARTITION rows.

Validation

  • ruff check, ruff format --check, mypy, and python -m compileall -q transfer_queue tutorial tests passed.
  • Added unit and Ray controller regression tests (TestMetadataInvariants in tests/test_controller_data_partitions.py and test_controller_metadata_and_slot_ownership_invariants in tests/test_controller.py), covering unregistered force_fetch/link fields, recycled slot rejection, pre-allocated slot set_custom_meta & promotion under TQ_PRE_ALLOC_SAMPLE_NUM > 1, mixed nested/non-tensor sample removal, and duplicate KV keys.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 22, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-22T14:07:02.519834Z ee33a15 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@0oshowero0
0oshowero0 force-pushed the fix/controller-metadata-invariants branch from fc93309 to 3dfb288 Compare November 22, 2026 13:18

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fc933099fa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread transfer_queue/controller.py Outdated
Comment on lines +1298 to +1301
unallocated = [idx for idx in global_indexes if idx not in partition.global_indexes]
if unallocated:
logger.error(f"Unallocated global indexes {unallocated} for partition {partition_id}")
return False

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Include reserved rows when checking partition ownership

When TQ_PRE_ALLOC_SAMPLE_NUM exceeds the number of inserted samples, get_metadata(..., mode="force_fetch") returns both active rows and still-reserved rows because it uses index_manager.get_indexes_for_partition. Passing that metadata to client.put now writes the payload but then fails the controller notification: reserved rows are absent from partition.global_indexes even though they are allocated to this partition. This rejects the entire batch, including its active rows. Check ownership against active plus pre-allocated indexes rather than treating reserved rows as recycled/unallocated; the new set_custom_meta filter also silently drops metadata for those reserved rows.

Useful? React with 👍 / 👎.

@0oshowero0
0oshowero0 force-pushed the fix/controller-metadata-invariants branch from 3dfb288 to ee33a15 Compare November 22, 2026 13:51
@0oshowero0

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ee33a154d9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread transfer_queue/controller.py Outdated
Comment on lines +1299 to +1300
allocated = partition.global_indexes | partition.pre_allocated_global_indexes
unallocated = [idx for idx in global_indexes if idx not in allocated]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Check ownership without copying the whole partition

For large or heavily preallocated partitions receiving small kv_put/kv_update batches, this union copies every active and reserved index on each write notification, even when only one row changes. That adds O(partition size) work and temporary memory to the controller's serial request path; ordinary puts repeat the copy in set_custom_meta. Check membership directly in the two existing sets in both locations so ownership validation scales with the requested batch rather than the entire partition.

Useful? React with 👍 / 👎.

@0oshowero0
0oshowero0 force-pushed the fix/controller-metadata-invariants branch from ee33a15 to 349ef0f Compare November 22, 2026 14:16
@0oshowero0
0oshowero0 force-pushed the fix/controller-metadata-invariants branch from 349ef0f to 3ae586a Compare November 22, 2026 14:30
@0oshowero0
0oshowero0 merged commit 8616a02 into Ascend:main Nov 22, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant