Repository navigation
OpenConceptLab/ocl_online#639 | dry_run_expansions compares production toggles with given toggles on, and simulates adding references, creating a version and importing references - #996
Conversation
…n toggles with given toggles on, and simulates adding references, creating a version and importing references
| try: | ||
| report = run() | ||
| except Exception as ex: # pylint: disable=broad-except | ||
| report = {'key': key, 'error': repr(ex)} |
There was a problem hiding this comment.
Failed items are skipped on resume. An item that raises is written to --output with its key, and get_done_keys then treats it as done. If the database or pgbouncer restarts mid-run, every remaining item fails with OperationalError, gets written as an error, and is skipped on every rerun, so the run never completes. Count only reports without error as done.
| if self.options['sample']: | ||
| queryset = ExpansionDryRun.get_heads_with_loose_locks() if self.options['loose_locks'] else \ | ||
| Collection.objects.filter(version='HEAD') | ||
| ids = list(queryset.exclude(expansion_uri=None).exclude(expansion_uri='').values_list('id', flat=True)) |
There was a problem hiding this comment.
--seed doesn't make samples repeatable. values_list('id') here, and head.references.filter(include=True) in run_add, have no ORDER BY, so random.sample picks from whatever order Postgres returns, and the same seed can sample different HEADs or references. Rebuild's --sample uses order_by('?'), which ignores the seed altogether. Add order_by('id') before sampling, and sample rebuild expansions in Python.
|
|
||
| @staticmethod | ||
| def get_head(uri): | ||
| head = Collection.objects.filter(uri=uri, version='HEAD').first() |
There was a problem hiding this comment.
HEAD URIs ending in /HEAD/ aren't found. A HEAD's uri has no HEAD/ segment, so the /orgs/ORG/collections/COLLECTION/HEAD/ form in this PR's description fails with CommandError here and in get_expansion. Either accept that form, for example by dropping a trailing HEAD/ before the lookup, or change the examples to /orgs/ORG/collections/COLLECTION/.
| run = self.dry_run.rebuild if self.dry_run.toggles else self.rebuild_against_stored | ||
| self.run_safely(expansion.uri, partial(run, expansion), self.write_summary) | ||
|
|
||
| def rebuild_against_stored(self, expansion): |
There was a problem hiding this comment.
The --env path isn't guarded. Without --toggle, rebuilds still run Expansion.dry_run_rebuild outside ExpansionDryRun.rolled_back, so Celery tasks aren't blocked, ES sync stays on and indexing isn't patched there, while every other mode has all three. Running this path through rolled_back too, with toggles read for --env, would give one guarded path.
| version = Collection( | ||
| mnemonic=head.mnemonic, version=self.DRY_RUN_VERSION, organization=head.organization, user=head.user, | ||
| released=True, autoexpand=True) | ||
| errors = Collection.persist_new_version(version, head.created_by, sync=True) |
There was a problem hiding this comment.
Row locks for the whole seed. persist_new_version takes select_for_update on HEAD and updates every sibling's is_latest_version, and the seed now runs inline in the same transaction, so those locks are held until the rollback. For a large collection that can be minutes, and writes to that HEAD or its versions block meanwhile, which matters on production. At least say so in the help.
| if not path or not os.path.exists(path): | ||
| return set() | ||
| with open(path) as file: | ||
| done = {json.loads(line).get('key') for line in file if line.strip()} |
There was a problem hiding this comment.
An old-format output file crashes here. --output used to write one JSON list. Pointing it at a file in that format (for example a .json file from an earlier run) makes json.loads fail on the first line, and the command stops with a traceback. Skip lines that don't parse, or raise a clear CommandError.
| def get_heads(self): | ||
| heads = [self.get_head(uri) for uri in self.options['uris']] | ||
| if self.options['sample']: | ||
| queryset = ExpansionDryRun.get_heads_with_loose_locks() if self.options['loose_locks'] else \ |
There was a problem hiding this comment.
--loose-locks applies to every mode. The help says it's for add, but get_heads applies it to the version and import samples too. Either reject it outside add, or update the help.
| return {'head': head.uri, **report} | ||
|
|
||
| @staticmethod | ||
| def count_newest_not_in_source_versions(head_expansion, expansion): |
There was a problem hiding this comment.
This count isn't tested. No test asserts the value of count_newest_not_in_source_versions. In the version test's fixture, two versions of c1 are marked is_latest_version, and the count was 1 both off and on. This is the number #586 asks for, so a regression here would go unnoticed. Add a fixture with a single latest version, and assert 0 with the toggle off and 1 with it on.
| if not source: | ||
| raise CommandError(f'No source found for {self.options["source"]}') | ||
| for head in heads: | ||
| expressions = ExpansionDryRun.get_import_expressions(head, source, self.options['size']) |
There was a problem hiding this comment.
Work done before the skip check. get_import_expressions loads and sorts every active concept URI of the source, once per HEAD, before checking whether the item is already done. Load the source's URIs once per run, and do the per-HEAD exclusion and sampling after the skip check.
…y failed items on resume, seeded samples repeat, HEAD/ URIs, guarded --env rebuild, one source load per import
|
Review fixes in 201895f:
|
snyaggarwal
left a comment
There was a problem hiding this comment.
Verdict: ready to merge after one fix. I recommend fixing the --loose-locks sampling first; the other three can follow.
I checked the 9 findings from the first review against 201895f, and all are addressed. This pass found 4 more:
- Medium:
--loose-locksmisses HEADs whose value-set lock is on an older release. With the toggle off, value-set locks are ignored entirely, so those HEADs are affected too. This matters for the #586 add-path sample, so it's worth fixing before that runs. - Low: a HEAD whose
expansion_uripoints to a deleted expansion crashes the whole command. - Low: an
ENVthat isn't a toggles column makes everyToggle.getraise. - Low: a resumed import run samples differently from an uninterrupted one.
The rollback guard itself checks out: tasks are blocked before celery_once's Redis lock, ES sync and expansion indexing are off, and on_commit callbacks are dropped on rollback. Version creation going through the real create path, run inline, is the right call.
The verdict is posted as a comment review because GitHub doesn't let a PR's author approve it.
🤖 Generated with Claude Code
…s: value set locks in --loose-locks, HEADs without an expansion, unknown env toggles, per-item seeded import samples
|
Second review fixes in 46c2b16:
|
snyaggarwal
left a comment
There was a problem hiding this comment.
Verdict: ready to merge after one fix.
I checked the 4 findings from the second review against 46c2b16, and all are addressed. This pass found 1 more, the same sampling gap on the source side:
- Medium:
--loose-locksmisses HEADs whose source lock is on an older release, for example a HEAD locked to v1 after v2 is released. A new reference resolves throughresolve_system_version, which uses the lock only with the toggle on, so those HEADs are exactly where the toggle changes what an add gets.test_add_references_resolves_within_locked_versionand this PR's own add test both show it. Fix it before the #586 add sample runs.
Nothing else: the rollback guard, resuming, seeding, HEAD URI handling and toggle columns all check out on this pass.
🤖 Generated with Claude Code
| loose = Q(version='HEAD') | Q(retired=True) | Q(released=False) | ||
| head_expansions = Expansion.objects.filter(collection_version__version='HEAD') | ||
| expansion_uris = { | ||
| *head_expansions.filter(evaluated_source_versions__in=Source.objects.filter(loose)).values_list( |
There was a problem hiding this comment.
Misses source locks on older releases. A new reference resolves through CollectionReference.resolve_system_version, and that uses the expansion's source locks only with the lock toggle on. So a HEAD locked to v1 after v2 is released gets v2 with the toggle off and v1 with it on. That's the most common lock, and this filter leaves it out because v1 is released and not retired. Use the same rule for sources as for value sets: any locked version that isn't its repo's latest release. If you compare against is_latest_version, check what that flag means for an unreleased newer version, or the sample will also pick locks that the toggle doesn't change.
…les HEADs locked at any version but their repo's latest release
… matches its sampling
|
Third review fix in 7e75ad4, help text in 2f02d1f:
|
snyaggarwal
left a comment
There was a problem hiding this comment.
Verdict: approve. This is ready to merge; the one finding below is a nit and doesn't need to block it.
I checked the third review's fix against 2f02d1f, and it's addressed. --loose-locks now picks HEADs locked to any source or value-set version other than their repo's latest release, and that matches how a versionless reference resolves with the toggle off (ReferenceResolution.version → get_latest_released_version).
I also checked for anything the earlier passes might have missed. Every task in core is built on AsyncTask or QueueOnceCustomTask, so the guard blocks them all before celery_once's Redis lock is taken. No base=QueueOnce or shared_task task gets past it.
- Nit: the latest-release rule leaves out a HEAD flagged released, but
get_latest_released_versiondoesn't. Only repos with a released HEAD are affected (inline comment).
The verdict is posted as a comment review because GitHub doesn't let a PR's author approve it.
🤖 Generated with Claude Code
| latest = {} | ||
| releases = klass.objects.filter( | ||
| mnemonic__in={mnemonic for mnemonic, _, _ in repos}, is_active=True, released=True, retired=False | ||
| ).exclude(version='HEAD').values_list('id', 'mnemonic', 'organization_id', 'user_id', 'created_at') |
There was a problem hiding this comment.
Nit: HEAD is excluded here but not in resolution. A versionless reference resolves through ReferenceResolution.version to get_latest_released_version(), which keeps a HEAD flagged released. For a repo whose released HEAD is newer than its last real release, a lock at that release does change with the toggle but isn't sampled, and a lock at HEAD is sampled but doesn't change. That's rare. Dropping .exclude(version='HEAD') would match how resolution picks.
OpenConceptLab/ocl_online#639
dry_run_expansionscould only rebuild expansions, compared with what's stored. Against a staging database where every new toggle is already on, that can't show what one toggle does on its own, and it can't simulate adding references or creating a version.What changes
--toggle NAME(repeatable): each change runs twice, with production's toggle values and then with these toggles on as well, and the two results are compared. An unknown toggle name is an error. Without--toggle, rebuilds work as before (compared with the stored expansion,--envstill applies), now inside the same guard.--mode:rebuild(default): rebuilds the expansion.add: removes each sampled reference and its members, then adds the reference again as new.--references Nper HEAD,--sample Nrandom HEADs,--loose-locks(add only) to sample only HEADs locked to a source or value set version that isn't its repo's latest release, where the lock toggle changes what an add gets.version: creates a release of each HEAD through the version create path, run inline, and compares it with HEAD's expansion:built_with, source versions, members against HEAD, and HEAD concepts whose newest version isn't in the version's source version. Version creation locks the HEAD row, so writes to that HEAD wait until each run rolls back.import: bulk imports--sizeversionless concepts of--sourcethe way the importer adds references, timed off, on, on, off.--seedmakes sampling repeatable.--outputnow appends one JSON line per item as it finishes; a rerun skips items already in the file, except failed ones. HEAD URIs work with or without a trailingHEAD/.core/collections/dry_runs.py. Every run is in a transaction that is rolled back; Celery tasks are blocked and counted, not queued; ES auto-sync and expansion indexing are off.Commands
Rebuild with one toggle on, against production's toggles:
Rebuild a random sample with two toggles on together:
Re-add references under the lock toggle, on HEADs where the lock matters plus named HEADs:
Create a version of each HEAD under the lock toggle:
Import benchmark, 100 and then 500 concepts:
Rebuild against the stored expansion, as before:
Each item prints the members that change per reference, source and collection version changes, and the two timings. The last line lists the tasks that weren't queued. Rerunning a command with the same
--outputpicks up where it stopped.Testing
--output.core.collectionstests pass locally;pylint -j0 core/passes.🤖 Generated with Claude Code