Skip to content

OpenConceptLab/ocl_online#639 | dry_run_expansions compares production toggles with given toggles on, and simulates adding references, creating a version and importing references - #996

Merged
snyaggarwal merged 5 commits into
masterfrom
ocl_online#639
Nov 18, 2026

Conversation

@snyaggarwal

@snyaggarwal snyaggarwal commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

OpenConceptLab/ocl_online#639

dry_run_expansions could only rebuild expansions, compared with what's stored. Against a staging database where every new toggle is already on, that can't show what one toggle does on its own, and it can't simulate adding references or creating a version.

What changes

  • --toggle NAME (repeatable): each change runs twice, with production's toggle values and then with these toggles on as well, and the two results are compared. An unknown toggle name is an error. Without --toggle, rebuilds work as before (compared with the stored expansion, --env still applies), now inside the same guard.
  • --mode:
    • rebuild (default): rebuilds the expansion.
    • add: removes each sampled reference and its members, then adds the reference again as new. --references N per HEAD, --sample N random HEADs, --loose-locks (add only) to sample only HEADs locked to a source or value set version that isn't its repo's latest release, where the lock toggle changes what an add gets.
    • version: creates a release of each HEAD through the version create path, run inline, and compares it with HEAD's expansion: built_with, source versions, members against HEAD, and HEAD concepts whose newest version isn't in the version's source version. Version creation locks the HEAD row, so writes to that HEAD wait until each run rolls back.
    • import: bulk imports --size versionless concepts of --source the way the importer adds references, timed off, on, on, off.
  • --seed makes sampling repeatable. --output now appends one JSON line per item as it finishes; a rerun skips items already in the file, except failed ones. HEAD URIs work with or without a trailing HEAD/.
  • The runner lives in core/collections/dry_runs.py. Every run is in a transaction that is rolled back; Celery tasks are blocked and counted, not queued; ES auto-sync and expansion indexing are off.

Commands

Rebuild with one toggle on, against production's toggles:

python manage.py dry_run_expansions /orgs/ORG/collections/COLLECTION/HEAD/ \
  --toggle INTENSIONAL_REFERENCE_RESOURCE_VERSIONS_TOGGLE --output /tmp/rebuild.jsonl

Rebuild a random sample with two toggles on together:

python manage.py dry_run_expansions --sample 25 --seed 576 \
  --toggle RESOURCE_VERSION_REPO_RESOLUTION_TOGGLE --toggle INTENSIONAL_REFERENCE_RESOURCE_VERSIONS_TOGGLE \
  --output /tmp/rebuild_sample.jsonl

Re-add references under the lock toggle, on HEADs where the lock matters plus named HEADs:

python manage.py dry_run_expansions /orgs/ORG/collections/COLLECTION/ \
  --mode add --toggle EXPANSION_LOCKED_REPO_VERSION_TOGGLE \
  --sample 15 --loose-locks --references 5 --seed 639 --output /tmp/639_add.jsonl

Create a version of each HEAD under the lock toggle:

python manage.py dry_run_expansions /orgs/ORG/collections/A/ /orgs/ORG/collections/B/ \
  --mode version --toggle EXPANSION_LOCKED_REPO_VERSION_TOGGLE --output /tmp/639_version.jsonl

Import benchmark, 100 and then 500 concepts:

python manage.py dry_run_expansions /orgs/ORG/collections/COLLECTION/ \
  --mode import --toggle EXPANSION_LOCKED_REPO_VERSION_TOGGLE --source /orgs/CIEL/sources/CIEL/ --size 100
python manage.py dry_run_expansions /orgs/ORG/collections/COLLECTION/ \
  --mode import --toggle EXPANSION_LOCKED_REPO_VERSION_TOGGLE --source /orgs/CIEL/sources/CIEL/ --size 500

Rebuild against the stored expansion, as before:

python manage.py dry_run_expansions /orgs/ORG/collections/COLLECTION/HEAD/ --env production

Each item prints the members that change per reference, source and collection version changes, and the two timings. The last line lists the tasks that weren't queued. Rerunning a command with the same --output picks up where it stopped.

Testing

  • Tests for each mode, the rollback and blocked tasks, the command's argument errors, and resuming from --output.
  • core.collections tests pass locally; pylint -j0 core/ passes.

🤖 Generated with Claude Code

…n toggles with given toggles on, and simulates adding references, creating a version and importing references
try:
report = run()
except Exception as ex: # pylint: disable=broad-except
report = {'key': key, 'error': repr(ex)}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Failed items are skipped on resume. An item that raises is written to --output with its key, and get_done_keys then treats it as done. If the database or pgbouncer restarts mid-run, every remaining item fails with OperationalError, gets written as an error, and is skipped on every rerun, so the run never completes. Count only reports without error as done.

if self.options['sample']:
queryset = ExpansionDryRun.get_heads_with_loose_locks() if self.options['loose_locks'] else \
Collection.objects.filter(version='HEAD')
ids = list(queryset.exclude(expansion_uri=None).exclude(expansion_uri='').values_list('id', flat=True))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--seed doesn't make samples repeatable. values_list('id') here, and head.references.filter(include=True) in run_add, have no ORDER BY, so random.sample picks from whatever order Postgres returns, and the same seed can sample different HEADs or references. Rebuild's --sample uses order_by('?'), which ignores the seed altogether. Add order_by('id') before sampling, and sample rebuild expansions in Python.


@staticmethod
def get_head(uri):
head = Collection.objects.filter(uri=uri, version='HEAD').first()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HEAD URIs ending in /HEAD/ aren't found. A HEAD's uri has no HEAD/ segment, so the /orgs/ORG/collections/COLLECTION/HEAD/ form in this PR's description fails with CommandError here and in get_expansion. Either accept that form, for example by dropping a trailing HEAD/ before the lookup, or change the examples to /orgs/ORG/collections/COLLECTION/.

run = self.dry_run.rebuild if self.dry_run.toggles else self.rebuild_against_stored
self.run_safely(expansion.uri, partial(run, expansion), self.write_summary)

def rebuild_against_stored(self, expansion):

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The --env path isn't guarded. Without --toggle, rebuilds still run Expansion.dry_run_rebuild outside ExpansionDryRun.rolled_back, so Celery tasks aren't blocked, ES sync stays on and indexing isn't patched there, while every other mode has all three. Running this path through rolled_back too, with toggles read for --env, would give one guarded path.

version = Collection(
mnemonic=head.mnemonic, version=self.DRY_RUN_VERSION, organization=head.organization, user=head.user,
released=True, autoexpand=True)
errors = Collection.persist_new_version(version, head.created_by, sync=True)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Row locks for the whole seed. persist_new_version takes select_for_update on HEAD and updates every sibling's is_latest_version, and the seed now runs inline in the same transaction, so those locks are held until the rollback. For a large collection that can be minutes, and writes to that HEAD or its versions block meanwhile, which matters on production. At least say so in the help.

if not path or not os.path.exists(path):
return set()
with open(path) as file:
done = {json.loads(line).get('key') for line in file if line.strip()}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An old-format output file crashes here. --output used to write one JSON list. Pointing it at a file in that format (for example a .json file from an earlier run) makes json.loads fail on the first line, and the command stops with a traceback. Skip lines that don't parse, or raise a clear CommandError.

def get_heads(self):
heads = [self.get_head(uri) for uri in self.options['uris']]
if self.options['sample']:
queryset = ExpansionDryRun.get_heads_with_loose_locks() if self.options['loose_locks'] else \

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--loose-locks applies to every mode. The help says it's for add, but get_heads applies it to the version and import samples too. Either reject it outside add, or update the help.

return {'head': head.uri, **report}

@staticmethod
def count_newest_not_in_source_versions(head_expansion, expansion):

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This count isn't tested. No test asserts the value of count_newest_not_in_source_versions. In the version test's fixture, two versions of c1 are marked is_latest_version, and the count was 1 both off and on. This is the number #586 asks for, so a regression here would go unnoticed. Add a fixture with a single latest version, and assert 0 with the toggle off and 1 with it on.

if not source:
raise CommandError(f'No source found for {self.options["source"]}')
for head in heads:
expressions = ExpansionDryRun.get_import_expressions(head, source, self.options['size'])

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Work done before the skip check. get_import_expressions loads and sorts every active concept URI of the source, once per HEAD, before checking whether the item is already done. Load the source's URIs once per run, and do the per-HEAD exclusion and sampling after the skip check.

…y failed items on resume, seeded samples repeat, HEAD/ URIs, guarded --env rebuild, one source load per import
@snyaggarwal

Copy link
Copy Markdown
Contributor Author

Review fixes in 201895f:

  • Failed items are retried on resume; only reports without error count as done.
  • Samples are taken from id-ordered lists (HEADs, expansions, references), so --seed repeats them.
  • HEAD URIs ending in HEAD/ resolve.
  • The --env rebuild path runs inside the same guard as the others, with toggles read for that env.
  • Version mode's help and docstring say HEAD stays locked until each run rolls back.
  • An --output file that isn't this command's JSON lines raises a CommandError.
  • --loose-locks is rejected outside --mode add.
  • newest_not_in_source_versions is asserted (0 off, 1 on) with a single-latest fixture.
  • Import loads the source's concept URIs once per run, after the skip check.

pylint -j0 core/ passes; core.collections tests pass (282).

@snyaggarwal snyaggarwal left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: ready to merge after one fix. I recommend fixing the --loose-locks sampling first; the other three can follow.

I checked the 9 findings from the first review against 201895f, and all are addressed. This pass found 4 more:

  • Medium: --loose-locks misses HEADs whose value-set lock is on an older release. With the toggle off, value-set locks are ignored entirely, so those HEADs are affected too. This matters for the #586 add-path sample, so it's worth fixing before that runs.
  • Low: a HEAD whose expansion_uri points to a deleted expansion crashes the whole command.
  • Low: an ENV that isn't a toggles column makes every Toggle.get raise.
  • Low: a resumed import run samples differently from an uninterrupted one.

The rollback guard itself checks out: tasks are blocked before celery_once's Redis lock, ES sync and expansion indexing are off, and on_commit callbacks are dropped on rollback. Version creation going through the real create path, run inline, is the right call.

The verdict is posted as a comment review because GitHub doesn't let a PR's author approve it.

🤖 Generated with Claude Code

Comment thread core/collections/dry_runs.py Resolved
Comment thread core/collections/dry_runs.py Resolved
…s: value set locks in --loose-locks, HEADs without an expansion, unknown env toggles, per-item seeded import samples
@snyaggarwal

Copy link
Copy Markdown
Contributor Author

Second review fixes in 46c2b16:

  • --loose-locks also samples HEADs with a value-set lock at any version but that value set's latest.
  • HEADs whose expansion_uri names no expansion are rejected when named, left out of samples, and skipped rather than crashing.
  • An env that isn't a toggles column reads every toggle as off, as Toggle.to_dict does.
  • With --seed, each import sample comes from its own generator seeded from the seed and the HEAD's URI, so a resumed run samples the same as an uninterrupted one.

pylint -j0 core/ passes; core.collections tests pass (285).

@snyaggarwal snyaggarwal left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: ready to merge after one fix.

I checked the 4 findings from the second review against 46c2b16, and all are addressed. This pass found 1 more, the same sampling gap on the source side:

  • Medium: --loose-locks misses HEADs whose source lock is on an older release, for example a HEAD locked to v1 after v2 is released. A new reference resolves through resolve_system_version, which uses the lock only with the toggle on, so those HEADs are exactly where the toggle changes what an add gets. test_add_references_resolves_within_locked_version and this PR's own add test both show it. Fix it before the #586 add sample runs.

Nothing else: the rollback guard, resuming, seeding, HEAD URI handling and toggle columns all check out on this pass.

🤖 Generated with Claude Code

Comment thread core/collections/dry_runs.py Outdated
loose = Q(version='HEAD') | Q(retired=True) | Q(released=False)
head_expansions = Expansion.objects.filter(collection_version__version='HEAD')
expansion_uris = {
*head_expansions.filter(evaluated_source_versions__in=Source.objects.filter(loose)).values_list(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Misses source locks on older releases. A new reference resolves through CollectionReference.resolve_system_version, and that uses the expansion's source locks only with the lock toggle on. So a HEAD locked to v1 after v2 is released gets v2 with the toggle off and v1 with it on. That's the most common lock, and this filter leaves it out because v1 is released and not retired. Use the same rule for sources as for value sets: any locked version that isn't its repo's latest release. If you compare against is_latest_version, check what that flag means for an unreleased newer version, or the sample will also pick locks that the toggle doesn't change.

@snyaggarwal

Copy link
Copy Markdown
Contributor Author

Third review fix in 7e75ad4, help text in 2f02d1f: --loose-locks now samples HEADs locked to any source or value set version that isn't its repo's latest release (newest active, released, unretired version other than HEAD). That covers a HEAD locked to v1 after v2 is released. It compares against the latest release, not is_latest_version, so a newer unreleased version doesn't make a lock look affected. The test covers source locks at an older release, the latest release and HEAD, and value-set locks at an older and the latest version.

pylint -j0 core/ passes; core.collections tests pass (285).

@snyaggarwal snyaggarwal left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: approve. This is ready to merge; the one finding below is a nit and doesn't need to block it.

I checked the third review's fix against 2f02d1f, and it's addressed. --loose-locks now picks HEADs locked to any source or value-set version other than their repo's latest release, and that matches how a versionless reference resolves with the toggle off (ReferenceResolution.version → get_latest_released_version).

I also checked for anything the earlier passes might have missed. Every task in core is built on AsyncTask or QueueOnceCustomTask, so the guard blocks them all before celery_once's Redis lock is taken. No base=QueueOnce or shared_task task gets past it.

  • Nit: the latest-release rule leaves out a HEAD flagged released, but get_latest_released_version doesn't. Only repos with a released HEAD are affected (inline comment).

The verdict is posted as a comment review because GitHub doesn't let a PR's author approve it.

🤖 Generated with Claude Code

latest = {}
releases = klass.objects.filter(
mnemonic__in={mnemonic for mnemonic, _, _ in repos}, is_active=True, released=True, retired=False
).exclude(version='HEAD').values_list('id', 'mnemonic', 'organization_id', 'user_id', 'created_at')

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: HEAD is excluded here but not in resolution. A versionless reference resolves through ReferenceResolution.version to get_latest_released_version(), which keeps a HEAD flagged released. For a repo whose released HEAD is newer than its last real release, a lock at that release does change with the toggle but isn't sampled, and a lock at HEAD is sampled but doesn't change. That's rare. Dropping .exclude(version='HEAD') would match how resolution picks.

@snyaggarwal
snyaggarwal merged commit 815c321 into master Nov 18, 2026
2 checks passed
@snyaggarwal
snyaggarwal deleted the ocl_online#639 branch November 18, 2026 04:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant