Skip to content

feat(pstack): hand verification maintenance back to its delivery owner - #781

Open
justin808 wants to merge 9 commits into
cursor:mainfrom
justin808:jg-codex/verification-maintenance-handoff
Open

justin808 wants to merge 9 commits into
cursor:mainfrom
justin808:jg-codex/verification-maintenance-handoff

Conversation

@justin808

@justin808 justin808 commented Oct 22, 2026 •

Copy link
Copy Markdown

🤖 Claude Code · Anthropic · claude-fable-5-2 · medium

Recommendation: Merge · 🔴 Risk: High · Code Walkthrough · Code Review · Checkpoint

Summary

/maintain-verification-skill now hands its result back when the request names a delivery owner. With no owner named it ships as before: one PR for proven corrections.

Intent

Why: The skill always opened its own PR. That competes with a task whose workflow already owns commits and publication.

Scope: Step 6 of the skill ("Ship or hand back"), one example prompt in the verify-and-ship guide, and the regenerated Claude Code copies. 4 files, +34 / −6.

Safe because: Step 6 branches on one thing, whether the request names a delivery owner. No owner named is the normal case, including unattended runs, and an installed workflow alone names no owner. Steps 1 to 5 are untouched.

Risk is high because the rule decides who may commit and open PRs, and a revert cannot retract an unintended publication. Review step 6 first.

Verification

  • Live runs on this wording against a real verification skill: with a named owner, 2 of 2 runs made no commit, branch, or PR; with none, 3 of 3 unattended runs opened exactly one PR. The same comment shows two looser wordings that each failed one direction.
  • node scripts/validate-plugins.mjs, node scripts/check-pstack-docs.mjs, and node scripts/generate-pstack-harness-trees.mjs --check pass locally at bd11d3a.
  • Clean, blocked, and installed-but-unassigned runs matched the rule on earlier wording.

Not verified: Samples are small and from one model. Clean and blocked outcomes were not repeated on bd11d3a, and one further standalone run was inconclusive. The hosted Validate plugins workflow needs a maintainer to approve the fork run. Locally, one statechart launcher test in cli.test.ts fails on every head tried (Node 24.8.0, macOS); 266 other script tests pass and 67 are skipped.

Rollback: Reverting restores unconditional standalone closeout. Nothing persists.

🤖 Generated with Claude Code

@justin808

This comment has been minimized.

@justin808 justin808 left a comment •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded — read the current walkthrough: #781 (review)

Walkthrough for commit `b7cba01`

🤖 Codex · OpenAI · gpt-6-astra (configured) · medium

Publisher model is configured; served model is UNKNOWN.

Code Walkthrough

Maintenance returns corrections and evidence when its caller explicitly owns delivery, while standalone runs still open one PR.

Closeout ownership

The changed outcome and closeout preserve source review, live coverage, edit scope and cleanup. Delegated runs return evidence even when clean or blocked. Installation alone does not assign ownership. The guide uses Shaka as an example; no Shaka dependency is added. Claude copies are generated from these sources.

Behavioral screen

The runner sends six cases to a tool-disabled model from an empty temporary Git workspace. It retains failures and continues the screen. The protocol keeps criteria out of model prompts and calls for independent grading. Final grading: 5/6 cases, 11/12 criteria. Clean coverage and artifact paths were reported, but its wording said nothing remained to hand off for delivery; that failed grade remains disclosed.

Validation, limits and rollback

Plugin validation, docs navigation and generated-tree checks passed. Script tests: 29 pass, 38 TLC-dependent skips, no failures. Six-case failure-retention and isolation controls passed. Independent technical review found no new issues on the final head. Four optional nits remain documented. This screen does not execute maintenance, publication or artifact retention; no end-to-end interoperability claim is made. Reverting restores unconditional closeout; no persistent state changes.

Walkthrough for commit b7cba01. This is a COMMENT, not an approval.

@justin808

This comment has been minimized.

@justin808
justin808 marked this pull request as draft October 22, 2026 21:49

@justin808 justin808 left a comment •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded — read the current walkthrough: #781 (review)

Walkthrough for commit `e6b64f4`

🤖 Codex · OpenAI · gpt-6.1-sol (configured) · medium

Served model unavailable in saved Codex logs.

Code Walkthrough

Maintenance returns corrections and evidence when its caller explicitly owns delivery, while standalone runs still open one PR.

Closeout ownership

The changed outcome and closeout preserve source review, live coverage, edit scope and cleanup. Delegated runs return evidence even when clean or blocked. Installation alone does not assign ownership. The guide uses Shaka as an example; no Shaka dependency is added. Claude copies are generated from these sources.

Behavioral screen

The runner captures six tool-disabled responses from an empty temporary Git workspace, retains failures, and withholds grading criteria. Earlier independent grading passed 5/6 cases and 11/12 criteria. After the clean-result clarification, the focused clean rerun explicitly returned its report and three evidence paths to Shaka; both criteria pass by author assessment. This is not a combined new six-case grade.

Validation, limits and rollback

Plugin, docs, generated-tree and whitespace checks passed. Earlier script tests: 29 pass, 38 TLC-dependent skips, no failures. Sixth independent technical review found no defects or risks; optional editorial nits remain documented. The screen measures stated closeout decisions, not actual publication restraint or artifact retention. The prior checkpoint covers b7cba01 and needs reassessment before merge readiness.

Walkthrough for commit e6b64f4. This is a COMMENT, not an approval.

@justin808

This comment has been minimized.

Brings the branch up to date with upstream pstack 0.15.36. The merge
had no conflicts; this PR's changes to maintain-verification-skill and
the verify-and-ship guide are unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* origin/main: (65 commits)
  feat(third_party): add Tesla plugin
  tailscale: drop the removed tag:grokbot sign-in from the README.
  tailscale: describe one-click personal tailnets and the bot computer path.
  X MCP guide: name edit_options in Posting step 4
  X MCP guide: pasted PIN, community posts, source regions, edit-then-post approval
  tailscale: drop the tag:grokbot sign-in and cover the bot computer path.
  X MCP guide: tighten post approval, broadcast update and 5xx backoff (from A/B eval)
  Fix Errors anchor in X MCP guide
  Trim X MCP guide skill to a minimal, additive reference
  X guide: a host's draft card is the approval for create_posts
  X plugin: posting with create_posts and tweet.write
  feat(lark-base): add the Lark Base REST MCP plugin
  Update the Gopuff logo
  Add Questrade plugin (third_party/questrade)
  business-central: set category to finance-and-legal
  Add delivery speed and more categories to Gopuff plugin keywords
  Update Gopuff plugin description, keywords and tags
  Add Business Central logo
  Add Microsoft Dynamics 365 Business Central plugin
  Add OnePay plugin (third_party/onepay)
  ...
justin808 added a commit to shakacode/shaka that referenced this pull request Nov 6, 2026
Shaka users who want pstack get a Setup page, docs/pstack.md, that
installs it beside Shaka in Cursor, Claude Code, or Codex and keeps it
current. The page gives Claude Code's third-party marketplace update
steps, a prompt that keeps pstack's maintenance results inside a Shaka
task, optional steps for trying the cursor/plugins#781 branch, and a
prompt for building a personal skill-pack updater. Working with Shaka
links the page, the sidebar lists it under Setup, and the changelog
records it.

Co-authored-by: Claude Opus 5.5 <[email protected]>
@justin808

This comment has been minimized.

@justin808

This comment has been minimized.

@justin808

Copy link
Copy Markdown
Author

🤖 Codex · OpenAI · gpt-6-astra (configured) · high

The committed-fixture rerun proves both changed-run closeouts at pstack 3cfd5adf9dfd7632f1066c30f41c6366e8869c5d: configured Shaka received a durable correction without worker publication, and standalone pstack created exactly one draft PR in an authorized private scratch repository.

Both isolated worktrees used the same committed fixture derived from flagship 1473c4b2106eb5cf9e96a9286c8f91819f7a108a, whose hosted verification passed. The existing verification skill came from merged flagship PR #87. The fixture changed the shared button selector from Add task to Create task; both runs reproduced the timeout and repaired precisely that line. The previously identified delete-recipe wording correction was already normalized in the common fixture.

Evidence Configured Shaka handoff Standalone
Ownership Shaka explicitly named delivery owner No delivery owner assigned
Source coverage Five feature readers Five feature readers
Before repair Create timed out; real failure artifacts retained Same failure and retained artifacts
After repair Board 11, create 8, update 6, delete 8, slow 8: all 41 checks passed Same five features and 41 passing checks
Correction One file, +1/-1, retained uncommitted at handoff Identical one-file, +1/-1 correction committed and pushed
Delivery Worker left HEAD/refs unchanged, with no delegated remote branch or PR; Shaka then committed and bound six actual receipts Worker created one draft PR against the committed fixture base; creation and all-state branch count verified
Final trees Identical corrected tree in both arms Identical corrected tree in both arms
Cleanup Owned processes exited; ports, socket and run DB released; external evidence retained Same cleanup checks passed; external evidence retained

Commands used the existing verify-flagship launch --mode static, doctor, broken drive create, reseed, repaired drive board|create|update|delete|slow, and cleanup. Delegated driving ran through generated private Shaka validation wrappers with shaka evidence run; Shaka subsequently ran evidence bind for the five live receipts and syntax receipt. All six bound to the owner commit's exact tested tree with unchanged material settings. Standalone used the helper directly, then committed, pushed its scratch branch and created the single draft PR. No technical review execution was added.

The handoff includes outcome, changed files, coverage and gaps, exact commands and observed results, and external evidence locations. The delegated correction was received and committed by Shaka, but was not published. Private repository links, machine paths and raw artifacts are withheld from this public report; they are retained with the local comparison for the requester.

Limits: one directed run per arm, not a blind benchmark or reliability estimate. Both used macOS, Ruby 3.4.9, Node 24.8.0 and static mode, with separate ports/databases. Source-reader concurrency was constrained by host capacity. Only delegated used private/local Shaka settings; these grant no team policy or merge authority. Three unrelated Dependabot PRs appeared in the scratch repository; the standalone branch has exactly one PR, and delegated has zero. The failed-create harness still exits before structured result/network JSON is written; actual logs, screenshots and ARIA evidence are retained. Ancillary bare-launch documentation may be stale based on source inspection and was not live-reproved or changed. Fresh clean/blocked outcome trials and TLC-dependent tests remain unverified; historical response-only screening is not relabeled as live proof.

Failed-drive residue evidence differs: delegated retained the failed browser process identities; standalone's process snapshot was taken after that browser exited. Standalone confirmed no remaining task browser residue, but does not provide a complete per-browser ownership ledger. Both have explicit server process ownership and successful final teardown.

The six-execution technical review ledger is unchanged. The upstream merge from e6b64f4 leaves the maintenance skill and behavioral screen unchanged. Validate plugins run 42131177509 remains action_required on the tested head, with no jobs. A maintainer must approve or rerun it and inspect the result. PR #781 remains unmerged with auto-merge off.

@justin808 justin808 left a comment •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded — read the current walkthrough: #781 (review)

Walkthrough for commit `3cfd5ad`

🤖 Codex · OpenAI · gpt-6-astra (configured) · high

Code Walkthrough

When a caller explicitly owns delivery, maintenance returns its corrections and proof; standalone runs retain one-PR publication.

Closeout ownership

The outcome and closeout separate verified corrections from shipped work. The caller receives outcome, files, coverage, commands, results and evidence, including clean or blocked runs. The guide demonstrates explicit Shaka ownership; installation alone changes nothing. Claude copies are generated from these sources. Source checks, live re-proofs, cleanup and edit scope stay required.

Behavioral evidence

The six-case runner isolates response-only cases in a temporary Git repository, withholds grading criteria, retains failures and continues the remaining cases. Its protocol limits claims to stated decisions. The committed-fixture live comparison proves configured Shaka handoff and exactly one standalone draft PR. Both arms passed41 checks; the six-case screen remains response-only evidence for other closeout decisions.

Validation, limits and rollback

Plugin, docs and generated-tree checks pass. Node 22 script suite: 267 passed, 67 TLC-dependent skips, zero failures using a canonical macOS temporary directory; the default-path symlink failure is retained. The six-execution technical review history ends at e6b64f4; no additional technical review ran. Reverting restores unconditional publication. Validate plugins needs upstream maintainer action. Merge and auto-merge remain off.

Walkthrough for commit 3cfd5ad. This is a COMMENT, not an approval.

@justin808

This comment has been minimized.

@justin808

justin808 commented Nov 11, 2026 •

Copy link
Copy Markdown
Author

🤖 Codex · OpenAI · gpt-6-astra (configured) · high

Fresh runtime cases now cover clean and blocked closeout both with an explicit Shaka owner and without one. A further changed run had Shaka installed and configured but unassigned; pstack published exactly one new private draft PR.

All cases used pstack 3cfd5adf9dfd7632f1066c30f41c6366e8869c5d and the existing flagship verification skill derived from verified flagship 1473c4b2106eb5cf9e96a9286c8f91819f7a108a / merged PR #87. The configured-unassigned run reused the original committed selector drift. Its private settings and wrappers were byte-identical to the earlier delegated configuration, and its final corrected tree matched both earlier changed runs.

Runtime case Live result Closeout observed
Explicit owner, changed (earlier) Five features, 41 checks pass Durable correction and proof handed to Shaka; worker published nothing; Shaka committed and bound six receipts
Unassigned, changed (earlier) Five features, 41 checks pass One private draft PR
Explicit owner, clean (new) Five features, 41 checks pass; no correction Report, coverage, commands, results and evidence handed to Shaka; no commit or PR
Explicit owner, blocked (new) Actual launch exit 1; source 5/5, live 0/5 Failed receipt and complete blocked report handed to Shaka; no correction or PR
Unassigned, clean (new) Five features, 41 checks pass; no correction Clean report and retained evidence; no branch, commit or PR
Unassigned, blocked (new) Actual launch exit 1; source 5/5, live 0/5 Blocked report and retained evidence; no branch, commit or PR
Explicit owner, blocked after starting (new) Board passed 11 checks; database-read fault then blocked doctor and the next drive Partial evidence and failed receipts handed to Shaka; all started resources cleaned; no correction or PR
Installed/configured, unassigned, changed (new) Original selector timeout reproduced, then five features and 41 checks pass Worker committed, pushed and created exactly one additional private draft PR; no owner handback

The two clean cases began from the same corrected committed fixture. Before either trial, its obsolete bare-launch warning was normalized against current source; that preparation is not counted as clean-run work. Each case performed its own five-reader source wave and live pass. Each subsequent blocked case explicitly reused those unchanged source findings.

The two pre-launch blocked cases used a comment-only ignored .env sentinel, which exercised the real helper's documented launch guard. These are controlled prerequisite failures, not organic regressions or verified-unreachable features. Launch refused before creating an instance, database, socket or listener. No unhealthy instance was driven, no nonexistent cleanup was claimed, and the test setup removed each sentinel only after receiving the blocked report.

A further owned case exercised blocking after real progress. After a healthy doctor, 11 passing board checks and reseeding, test setup removed read permission from only that run's disposable SQLite file. The actual doctor and the next drive's health gate failed; the other four features stayed unverified. The worker preserved partial evidence and cleaned up every started process, port, socket and run database before handing back the blocked report. No product or verification source changed.

Commands: verify-flagship launch --mode static, doctor, serial drive board|create|update|delete|slow, and cleanup; changed runs also reproduced failed drive create and performed reseed before re-proof. Owned cases used actual shaka evidence run receipts; the owner checked the failed receipt and bound the clean live receipts to the unchanged fixture commit. Clean/blocked Git and head-filtered publication snapshots confirm no task publication. The configured-unassigned run's branch PR count was 0→1 and repository count 4→5; the earlier trial PR and three Dependabot PRs were preserved.

Successful and partial-run teardown verified owned PID/start-time identities, free ports, absent socket/run database and retained artifacts. All 377 external evidence files remained hash-identical after the four new scratch worktrees were removed. Private paths, repository links and raw artifacts are withheld here; the requester has the compact local comparison and durable evidence index.

Limits: these are directed contract exercises, not a blind evaluation or reliability estimate. macOS/static mode only; source-reader concurrency varied with capacity. The configured target's private settings and access were healthy; an unrelated cross-host installation-pointer warning was retained without changing installed pins. Existing failed-create structured-JSON omission and 67 TLC-dependent skips remain documented; neither is claimed fixed. The prior response-only screen remains separate evidence.

No additional technical review ran; the six-execution ledger is unchanged. PR head remains pinned and the upstream merge left maintenance and screen files unchanged. Validate plugins remains action_required with zero jobs on this head. A maintainer must approve or rerun it and inspect the result. Merge and auto-merge remain off.

@justin808

This comment has been minimized.

Remove the response-only closeout screen under evals/, which added a
new top-level directory and Ruby to the repository; live runs recorded
on the PR cover the same cases. Fold the clean-run handoff rule into
step 6 and shorten it. Replace the Shaka-specific guide example with a
vendor-neutral prompt. Regenerate the Claude Code copies.

Co-Authored-By: Claude Fable 5.2 <[email protected]>
@justin808

justin808 commented Nov 14, 2026 •

Copy link
Copy Markdown
Author

Local Adversarial Review

Reviewed revision: bd11d3a

Outcome: No open findings; closed and optional findings are in history.

Coverage: openai/codex: Read-only source lookup permitted; actual inspection coverage is UNKNOWN. Test execution is prohibited.

Usage and attribution in the PR description.

Reviewed commits:

Commit Findings
90de508 openai/codex: 0
bd11d3a openai/codex: 0
Findings, assessments, and execution metadata

Finding details

Execution metadata

Round Commit Reviewer Findings Tokens Cost
1 90de508 openai/codex
model UNKNOWN
effort UNKNOWN
0 90,982 input
77,056 cached input
0 cache writes
1,051 output
128 reasoning output
$0.05 estimated
2 bd11d3a openai/codex
model UNKNOWN
effort UNKNOWN
0 136,337 input
113,152 cached input
0 cache writes
1,155 output
77 reasoning output
$0.07 estimated

Prompt: Shaka default · criteria not supplied

Commit 90de508 · openai/codex · 0 findings

Commit: 90de508 Trim the maintenance handoff change to the skill and guide

Review coverage:

Coverage excerpts may be incomplete or misidentified; inspect the original reports.

  • openai/codex: Read-only source lookup permitted; actual inspection coverage is UNKNOWN. Test execution is prohibited.

Unverified report excerpt · openai/codex:

Repository criteria: supplied — user-provided AGENTS.md agreements. No additional trusted base repository criteria were supplied.

Inspected the pinned maintenance skill, its Claude Code copy, both guide diffs, and unchanged references in the README, generator, help index, and customization guide.

Source inspection supports the stated goal: explicit delivery ownership triggers handoff for changed, clean, and blocked outcomes; standalone publication rules remain intact. Source/live checks, evidence retention, cleanup, and edit boundaries remain required. No credential additions or weakened trust gates were identified.

No behavioral execution results were supplied. `git diff --check` produced no diagnostics; a subsequent guide diff command failed because the sandbox prohibited a temporary file. Candidate code was not executed.

Observation — verification remains unproven at the workflow boundary. Prioritize these checks:

1. Explicit owner retention with changed, clean, and blocked outcomes: expect reports and retained evidence in all cases, with no maintenance-owned commits or PRs.
2. Standalone runs, including an installed but unassigned workflow: expect exactly one PR for changed and none for clean or blocked.
3. Repeat the owner-retained changed case against the base: expect its unconditional PR instruction to expose the regression that this change addresses.

No supplied test establishes failure before and success after.
Round 1 · 90de508 · openai/codex · effort UNKNOWN · 0 findings

Coverage

Repository criteria: supplied — user-provided AGENTS.md agreements. No additional trusted base repository criteria were supplied.

Inspected the pinned maintenance skill, its Claude Code copy, both guide diffs, and unchanged references in the README, generator, help index, and customization guide.

Source inspection supports the stated goal: explicit delivery ownership triggers handoff for changed, clean, and blocked outcomes; standalone publication rules remain intact. Source/live checks, evidence retention, cleanup, and edit boundaries remain required. No credential additions or weakened trust gates were identified.

No behavioral execution results were supplied. git diff --check produced no diagnostics; a subsequent guide diff command failed because the sandbox prohibited a temporary file. Candidate code was not executed.

Observation — verification remains unproven at the workflow boundary. Prioritize these checks:

  1. Explicit owner retention with changed, clean, and blocked outcomes: expect reports and retained evidence in all cases, with no maintenance-owned commits or PRs.
  2. Standalone runs, including an installed but unassigned workflow: expect exactly one PR for changed and none for clean or blocked.
  3. Repeat the owner-retained changed case against the base: expect its unconditional PR instruction to expose the regression that this change addresses.

No supplied test establishes failure before and success after.

Findings

no findings

REVIEWED 90de508 BY openai/codex EFFORT UNKNOWN FINDINGS 0

Commit bd11d3a · openai/codex · 0 findings

Commit: bd11d3a Make step 6 an explicit owner-named or no-owner branch

Review coverage:

Coverage excerpts may be incomplete or misidentified; inspect the original reports.

  • openai/codex: Read-only source lookup permitted; actual inspection coverage is UNKNOWN. Test execution is prohibited.

Unverified report excerpt · openai/codex:

Repository criteria: supplied — user-provided AGENTS.md instructions. No trusted base-repository AGENTS.md criteria were supplied.

Inspected the exact pinned diff across all four changed files, the complete maintenance skill, its generation contract, maintenance references, and the harness generation documentation. Source and generated copies contain matching behavioral changes. No candidate code was executed or files edited.

Supplied execution evidence consists of narrative reports that earlier wording stalled standalone runs or allowed owner-retained publication; no transcripts, commands, artifacts, or final-head results were supplied. Prior rounds reported no findings.

The revised instructions explicitly cover both delivery branches and preserve clean/blocked handoffs, evidence retention, live verification, and edit scope. No concrete contract drift, credential addition, permission expansion, or material maintenance issue was identified.

Observation — verification remains incomplete: source inspection cannot establish that agents follow the new branch reliably. Prioritize final-head runs through the actual skill invocation:

- Owner-retained `changed`: return corrections and evidence; create no commit, branch, or PR.
- Unattended standalone `changed`, including an installed but unassigned workflow: open exactly one PR without requesting ownership clarification.
- Owner-retained `clean` and `blocked`: return coverage, gaps, results, and surviving evidence; perform no publication.

Capture Git refs and publication activity alongside transcripts. Repeat against the base to establish that the owner-retained check detects the reverted behavior. No supplied test demonstrates that sensitivity.
Round 2 · bd11d3a · openai/codex · effort UNKNOWN · 0 findings

Coverage

Repository criteria: supplied — user-provided AGENTS.md instructions. No trusted base-repository AGENTS.md criteria were supplied.

Inspected the exact pinned diff across all four changed files, the complete maintenance skill, its generation contract, maintenance references, and the harness generation documentation. Source and generated copies contain matching behavioral changes. No candidate code was executed or files edited.

Supplied execution evidence consists of narrative reports that earlier wording stalled standalone runs or allowed owner-retained publication; no transcripts, commands, artifacts, or final-head results were supplied. Prior rounds reported no findings.

The revised instructions explicitly cover both delivery branches and preserve clean/blocked handoffs, evidence retention, live verification, and edit scope. No concrete contract drift, credential addition, permission expansion, or material maintenance issue was identified.

Observation — verification remains incomplete: source inspection cannot establish that agents follow the new branch reliably. Prioritize final-head runs through the actual skill invocation:

  • Owner-retained changed: return corrections and evidence; create no commit, branch, or PR.
  • Unattended standalone changed, including an installed but unassigned workflow: open exactly one PR without requesting ownership clarification.
  • Owner-retained clean and blocked: return coverage, gaps, results, and surviving evidence; perform no publication.

Capture Git refs and publication activity alongside transcripts. Repeat against the base to establish that the owner-retained check detects the reverted behavior. No supplied test demonstrates that sensitivity.

Findings

no findings

REVIEWED bd11d3a BY openai/codex EFFORT UNKNOWN FINDINGS 0

REVIEWED bd11d3a BY openai/codex EFFORT UNKNOWN FINDINGS 0

@justin808 justin808 left a comment •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded — read the current walkthrough: #781 (review)

Walkthrough for commit `90de508`

🤖 Claude Code · Anthropic · claude-fable-5-2 · medium

Code Walkthrough

A maintenance run now hands its result back when the user or calling workflow explicitly keeps delivery ownership. Before, it always opened its own PR for proven corrections, which competed with a task that already owned commits and publication.

Closeout rule

The changed outcome no longer promises a PR; it points to step 6. Step 6 branches once. With explicit ownership, the run returns its report and evidence and leaves commits and PRs to the owner, including for clean or blocked runs.

Without explicit ownership the old behavior holds: one PR for changed, none for clean or blocked. An installed workflow does not count as an owner. Steps 1 to 5 are untouched.

Guide and generated copies

The guide gains one vendor-neutral example prompt. The two files under harness/pstack/claude-code/ are generator output of these sources; generate-pstack-harness-trees.mjs --check reports them up to date.

Validation, risk, and rollback

Plugin validation, the docs check, and the generated-tree check pass locally. Script tests: 266 pass, 67 skipped, 1 fail; that statechart launcher test fails identically on the previous head on this machine.

Live runs on the previous head 3cfd5ad covered owned and standalone cases with longer step 6 wording and were not repeated here. The risk is a model misreading an ownership statement; the standalone path is the fallback. Reverting restores unconditional publication.

Walkthrough for commit 90de508. This is a COMMENT, not an approval.

@justin808

justin808 commented Nov 14, 2026 •

Copy link
Copy Markdown
Author

🤖 Codex · OpenAI · gpt-6.1-sol (configured) · medium

Superseded — read the current checkpoint: #781 (comment)

Earlier checkpoint

Checkpoint for 90de508

Recommendation: Merge

The trimmed change is one explicit ownership branch in step 6 plus a neutral guide example, the fresh review found nothing, and this checkpoint raised no concern.

Next: cursor/plugins maintainer approves the Validate plugins workflow run for this head.

Reviewer verdict: Proceed, no unresolved concerns.

Verification: Adequate, with gaps the reviewer judged acceptable.

  • Weakest claim: The shorter final wording preserves the ownership behavior demonstrated on 3cfd5ad; live checks did not exercise 90de508.
  • Next check: Repeat paired owned and standalone changed runs using the pinned final skill.

Risk: High. High risk is meant for a person to merge after a careful review that starts with the verification: does the evidence prove the change works, before the code?

  • The instruction changes who controls commits, PR publication, and merging.
  • An unintended publication can disclose content that reverting the instruction cannot retract.

With merge.auto_merge_max_risk: low, an agent merges a low rating; a person merges the rest.

Supporting analysis and execution details

Reviewer summary: Maintenance can return corrections to an explicitly assigned delivery owner while standalone runs retain one-PR delivery. Conflict frequency is unknown, and live behavioral evidence predates the final wording.

pstack/skills/maintain-verification-skill/SKILL.md resolves conflicting publication ownership through an explicit handoff while preserving standalone delivery.

pstack/docs/guide/06-verify-and-ship.md explains the choice without vendor coupling. Four changed files cover two source documents and their generated copies; no runtime dependency remains.

Review churn removed the oversized eval approach. Earlier live runs exercised owned, standalone, unassigned, clean, and blocked outcomes; final-head structural checks passed.

Alternative considered: A caller-only override would avoid shared instruction growth but leave the skill's unconditional publication requirement unresolved. The explicit ownership branch is the smallest coherent shared rule.

observed model: UNKNOWN; configured model: gpt-6.1-sol · recorded effort: medium · prompt: Shaka default

Native usage (openai · gpt-6.1-sol · UNKNOWN · medium), 1 response: 19,662 input, 12,160 cached input, 0 cache writes, 715 output, 325 reasoning output, 20,377 total.

Ruby checked this report's format and commit; the reviewer judged value, not correctness. Required checks, reviews, and merge authority still apply.

Squash commit message

Squash commit message for 90de508. Paste the title and the body into GitHub's squash merge boxes.

pstack: hand verification maintenance back to its delivery owner (#781)
maintain-verification-skill always opened its own PR for proven
corrections, which conflicts with a task whose workflow already owns
commits and publication. Step 6 becomes "Ship or hand back": when the
user or calling workflow explicitly retains delivery ownership, the run
returns the outcome, changed files, coverage, commands with observed
results, and evidence paths to that owner and leaves commits and PRs to
it. Clean and blocked runs still return their report. Standalone runs
are unchanged: one PR for changed, none for clean or blocked. An
installed workflow alone does not assign ownership. The verify-and-ship
guide gains one example prompt, and the Claude Code copies are
regenerated.

Co-authored-by: Claude Fable 5.2 <[email protected]>

@justin808 justin808 changed the title pstack: hand verification maintenance back to its delivery owner feat(pstack): hand verification maintenance back to its delivery owner Nov 14, 2026
Live runs on the previous wording showed an unattended standalone run
stopping without a PR because no owner had been named. A default-first
rewording fixed that but let a run with a named owner publish anyway.

Step 6 now checks the request for a named delivery owner, with examples
of what counts, and branches once: owner named means hand back with no
commit, branch, or PR; no owner named means ship, which is the normal
case including unattended runs. Regenerate the Claude Code copies.

Co-Authored-By: Claude Fable 5.2 <[email protected]>
@justin808

Copy link
Copy Markdown
Author

🤖 Claude Code · Anthropic · claude-fable-5-2 · medium

Live runs on the final step 6 wording (bd11d3a) held in both directions: two runs with a named owner published nothing, and three standalone runs each opened exactly one PR. Two earlier wordings each failed one direction, which is why step 6 is now an explicit branch.

Step 6 wording Owner named No owner named, unattended
Upstream main (unconditional) not applicable 1 of 1 opened one PR
90de508: hand-back condition first 1 of 1 handed back, nothing published 0 of 1: fixed the drift, then stopped without a PR because no owner was named
Uncommitted trial: default first 0 of 1: opened a PR despite the named owner 2 of 2 opened one PR
bd11d3a: explicit owner-named / no-owner branch 2 of 2 handed back; no commit, branch, or PR 3 of 3 opened one PR; 1 further run inconclusive
Method

Each run was a fresh headless Claude Code session (Claude Sonnet 5.5, permission checks on, no personal instructions or plugins) in a private disposable repository holding the React on Rails flagship demo and its verify-flagship skill from react-on-rails-demo-flagship#87. The fixture's driver clicks a button named Create task; the app's button is Add task.

The prompt asked for /maintain-verification-skill, pointed at the skill file under test, and said the run was unattended. Owner-named runs added one sentence: "Shaka owns delivery for this task." Nothing in any prompt mentioned commits or PRs.

Every run launched the app, drove all five mapped features, reproduced the create and delete timeouts, repaired the selector, re-drove, and cleaned up. After each run I compared local refs before and after, read the working tree, and listed the repository's branches and PRs.

What was checked on `bd11d3a`

Owner named, both runs: HEAD and all refs unchanged, the correction left uncommitted, no new remote branch or PR, and a closeout that returned the changed files, coverage, commands, results, and evidence path.

No owner named, three runs: one new branch pushed and exactly one PR opened each time, with the PR link in the closeout. One said: "No delivery owner was named, so I shipped it."

The inconclusive run prepared to ship, found the identical fix already open from an earlier test run of mine, and opened no duplicate. I closed the leftover test PRs before the last two runs. The file tested was byte-identical to the committed skill.

Limits

Small samples from one model on macOS; these are directed checks, not a reliability estimate. Clean and blocked outcomes were not repeated on this head; the earlier runs covered them on older wording. Ownership was asserted by one sentence in the prompt, not by a configured workflow. Private repository links and raw artifacts are withheld.

@justin808 justin808 left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Claude Code · Anthropic · claude-fable-5-2 · medium

Code Walkthrough

A maintenance run now hands its result back when the request names a delivery owner, and ships its own PR when it names none. Before, it always opened its own PR, which competed with a task that already owned commits and publication.

Closeout rule

The changed outcome no longer promises a PR; it points to step 6. Step 6 first checks the request for a named delivery owner and gives two examples of what counts. Then it branches once.

Owner named: return the report and evidence, and make no commit, branch, or PR. Clean and blocked runs still return their report. No owner named: ship without asking, which is the old behavior. Steps 1 to 5 are untouched.

Guide and generated copies

The guide gains one vendor-neutral example prompt. The two files under harness/pstack/claude-code/ are generator output of these sources; generate-pstack-harness-trees.mjs --check reports them up to date.

Validation, risk, and rollback

Plugin validation, the docs check, and the generated-tree check pass locally. Script tests: 266 pass, 67 skipped, 1 fail; that statechart launcher test fails identically on earlier heads on this machine.

The branch shape comes from live runs: two looser wordings each failed one direction. The risk is a model misreading whether an owner was named. Reverting restores unconditional publication.

Walkthrough for commit bd11d3a. This is a COMMENT, not an approval.

6. **Ship or stop.** For changed: one PR of proven corrections, re-read every changed file first. For clean or blocked: no PR, report the outcome and the coverage honestly.
6. **Ship or hand back.** Re-read every changed file first. Then check the request for a named delivery owner: a statement from the user or the calling workflow that someone else owns delivery, commits, or publication for this task, such as "X owns delivery" or "don't open a PR".
- **Owner named: hand back.** Return the outcome, changed files, coverage and gaps, exact commands with observed results, and retained evidence paths to that owner. Make no commit, branch, or PR; the owner does that. A clean or blocked run still returns its report and evidence; say that no publication is needed, not that no handoff is needed.
- **No owner named: ship.** This is the normal case, including unattended runs; do not ask or wait for an owner. For changed, open one PR of proven corrections and report its link. For clean or blocked, open no PR and report the outcome and coverage honestly.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Claude Code · Anthropic · claude-fable-5-2 · medium * walkthrough

"Including unattended runs; do not ask or wait" is deliberate. Without it, one unattended standalone run fixed the drift and then stopped because nobody had named an owner.

@justin808

Copy link
Copy Markdown
Author

🤖 Codex · OpenAI · gpt-6.1-sol (configured) · medium

Checkpoint for bd11d3a

Recommendation: Merge

Live runs on the final wording held in both directions, the review found nothing, and this checkpoint raised no concern.

Next: cursor/plugins maintainer approves the Validate plugins workflow run for this head.

Reviewer verdict: Proceed, no unresolved concerns.

Verification: Adequate, with gaps the reviewer judged acceptable.

  • Weakest claim: Clean and blocked runs preserve their reporting and handoff behavior under the final wording; those outcomes were exercised only on earlier wording.
  • Next check: Repeat an owner-named blocked run on bd11d3a and confirm retained evidence, a complete handoff, and no publication.

Risk: High. High risk is meant for a person to merge after a careful review that starts with the verification: does the evidence prove the change works, before the code?

  • The instructions govern authority to commit, create branches, and publish PRs.
  • A revert cannot retract information already published by an incorrect ownership decision.
  • Final-wording live runs directly exercised both publication choices, limiting the remaining uncertainty.

With merge.auto_merge_max_risk: low, an agent merges a low rating; a person merges the rest.

Supporting analysis and execution details

Reviewer summary: Maintenance can return verified corrections to an explicitly named delivery owner while standalone runs still publish one PR. Final-wording runs support both branches, although evidence covers one model and leaves clean and blocked outcomes tested only on earlier wording.

Open pstack/skills/maintain-verification-skill/SKILL.md first: explicit ownership prevents competing publication while preserving standalone delivery.

pstack/docs/guide/06-verify-and-ship.md explains the handoff without vendor coupling. Four changed files match the planned source and generated copies; no new runtime dependencies or configuration remain.

Review churn exposed ambiguous defaults. The final explicit branch passed both changed-run scenarios after two earlier formulations failed. The packet establishes no continuing approach problem; usage frequency and aggregate impact remain unknown.

Alternative considered: Keeping unconditional publication would preserve simpler wording but lose caller-controlled delivery. The observed owner-named runs justify the small instruction increase.

observed model: UNKNOWN; configured model: gpt-6.1-sol · recorded effort: medium · prompt: Shaka default

Native usage (openai · gpt-6.1-sol · UNKNOWN · medium), 1 response: 19,914 input, 12,160 cached input, 0 cache writes, 589 output, 160 reasoning output, 20,503 total.

Ruby checked this report's format and commit; the reviewer judged value, not correctness. Required checks, reviews, and merge authority still apply.

Squash commit message

Squash commit message for bd11d3a. Paste the title and the body into GitHub's squash merge boxes.

feat(pstack): hand verification maintenance back to its delivery owner (#781)
maintain-verification-skill always opened its own PR for proven
corrections, which conflicts with a task whose workflow already owns
commits and publication. Step 6 becomes "Ship or hand back" and branches
on whether the request names a delivery owner. Owner named: return the
outcome, changed files, coverage, commands with observed results, and
evidence paths, and make no commit, branch, or PR; clean and blocked
runs still return their report. No owner named: ship as before, one PR
for changed and none for clean or blocked, without asking, including on
unattended runs. An installed workflow alone names no owner. The
verify-and-ship guide gains one example prompt, and the Claude Code
copies are regenerated.

Co-authored-by: Claude Fable 5.2 <[email protected]>

@justin808
justin808 marked this pull request as ready for review November 16, 2026 09:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant