Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 102 additions & 0 deletions .agents/review/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# AI review

The author of a pull request reviews the branch with an AI agent before asking people to review it, fixes what the review finds or dismisses it with a reason, and says so in the pull request's "AI Review" section. The people who review the pull request then look at what an AI cannot judge, and do not run it through an AI themselves ("Working with it" below).

It works the same in every agent: the same prompts, the same record. The reviewers always start in a fresh context, without the conversation that produced the change, on a model no weaker than the author's ("Fresh reviewers" in `review.md`). The agent that wrote the change only takes their findings and settles them with the user.

These files come from the `ai-review` pack of your organisation's engineering-assets hub.

| File | What |
|---|---|
| [`review.md`](review.md) | the review: scope, directions, verifying and ranking findings, the record |
| [`directions/`](directions/) | one prompt per direction: what to look for and what counts as evidence |
| [`pure-refactor.md`](pure-refactor.md) | a check that a change claimed to keep behaviour keeps it |
| [`check.py`](check.py) | says whether the branch's review covers its last commit; a pre-push hook runs it |

## Directions

| Direction | Looks for |
|---|---|
| [Correctness](directions/correctness.md) | logic errors, edge inputs, swallowed and unnoticed errors, races, transactions, resources |
| [Security](directions/security.md) | secrets, injection, access checks, data exposure, insecure defaults, new dependencies |
| [Performance](directions/performance.md) | N+1 and unbounded queries, blocking calls in async code, work that grows with the data |
| [Design & Style](directions/design.md) | why each new thing exists, what already exists, layers, simplicity, naming, comments |
| [Tests](directions/tests.md) | new behaviour without a test, tests that check nothing, tests bent to fit |
| [Compatibility](directions/compatibility.md) | API and schema contracts, migrations, rollout, stored data, configuration, flags, stale docs |

A repository adds its own focus areas in a `## Review` section of `.agents/project.md`, one line each: `- <direction>: <what to check>`. A name that is none of the directions above becomes a direction of its own, with its own line in the pull request.

## Running it

The agent runs the review by itself once the last change of a task is committed (`AGENTS.md`), so nobody has to remember. Unlike the starter review, it is not read-only: it writes the record in `local/`, and the agent then fixes the findings the user agrees with. By hand:

| Agent | Review | Pure-refactor check |
|---|---|---|
| Claude Code | `/review-change` | `/review-change pure-refactor` |
| Codex | `$review-change` | `$review-change pure-refactor` |
| Cursor | `/review-change` | `/review-change pure-refactor` |
| OpenCode | "review my branch" | "check that my branch keeps behaviour" |
| any chat | paste `review.md`, the direction files and `git diff <base>...HEAD` (`<base>` as in `review.md`) | paste `pure-refactor.md` and the diff |

The request may name directions (`/review-change security performance`), ask for `full` (the whole branch again, not a follow-up over the changes since the last review), or give `staged`, `working` or a path.

The skill is `.agents/skills/review-change/`, where Codex, Cursor and OpenCode find it. In Claude Code, `/review-change` of the `claude` pack follows it whenever this directory exists, and the `reviewer` subagent is the fresh reviewer it starts for each direction.

## The record

Every run updates the branch's one record, `local/reviews/<branch>.md` (`local/` is gitignored): the commit the last complete run covered, the reviewers' model, the result per direction and each finding with its status: fixed, dismissed with the reason, or open. The next run reviews only what changed since that commit and does not bring back what was dismissed. The format is in [`review.md`](review.md).

## In the pull request

The description ends with an "AI Review" section, built from the record and nothing else, in the language `.agents/project.md` names for pull requests. With the `claude` pack, `/pr` adds it. Put the section into the repository's pull request template, so that people see it is expected:

```markdown
#### AI Review

Commit `a1b2c3d`, Claude Opus 5.5

- [x] Correctness - 2 fixed
- [x] Security - no findings
- [x] Performance - 1 fixed, 1 dismissed
- [x] Design & Style - 3 fixed
- [x] Tests - no findings
- [x] Compatibility - not applicable: no API, schema or configuration change

<details><summary>Dismissed (1)</summary>

- `src/keys.py:42` - N+1 when loading keys: the list holds at most 5 keys
</details>
```

- The first line is the record's `commit` (short sha) and `model`. When the user declined a follow-up over later fixes, it adds ", fixes after it not reviewed again".
- One line per direction of the record, in its order and under its name, the repository's own directions and Pure-refactor check included. A direction that was not run is unchecked: `- [ ] Tests - not run`.
- The block of dismissed findings lists each dismissed blocking or should-fix finding with its reason; no dismissed findings, no block. Nits never go there, and a finding about a secret names its place and kind, never the value.

## Working with it

- **The AI review is the author's job.** Before the pull request is opened for review (a draft opened only so that CI runs is not), the author reviews the branch, fixes what the review finds or dismisses it with a reason, and fills the "AI Review" section. A pull request without it, or with its boxes unchecked, is not ready for review. After the fixes, and after changes made in review, the author runs a follow-up on the new commits: it checks the fixes and the lines they touched, not the whole branch again. Commits that only fix findings may go without one: the author's call, and the section says so.
- **People review what an AI cannot judge**: does the change solve the task, are the model and the contracts right, is the rollout plan right, what does it do to its neighbours, will the next person understand it, and are the dismissed findings fair.
- **No AI-generated comments on someone else's pull request.** Do not run a colleague's change through an AI and post what it says. Using an AI to understand the code is fine; the comment is your own.
- **A bug the AI review should have caught** gets a review thread that starts with `ai-miss:`. Besides the fix, the author turns the miss into a rule where the review reads it, and links it in the reply: a line in the `Review` section of `.agents/project.md` if it is about this repository, otherwise a pull request or an issue to the hub on `packs/ai-review/.agents/review/directions/`.

## Wiring

Optional, once per repository, in its own files, in a pull request merged after the sync pull request that brings `.agents/review/`: until then the hook finds no script and fails every push. `check.py` needs Python 3.9 or later and only its standard library, so it runs in a repository of any language.

- A pre-push hook in `.pre-commit-config.yaml`. It never blocks a push, since unfinished branches are pushed to run CI, and only says when the review is missing or behind the branch:

```yaml
- repo: local
hooks:
- id: ai-review
name: ai review
entry: python .agents/review/check.py
language: python
pass_filenames: false
always_run: true
verbose: true
stages: [pre-push]
```

- The way the repository installs its hooks installs `pre-push` too. Plain `pre-commit install` does once `default_install_hook_types` lists it (with `commit-msg` where a hook needs it); a README or Makefile that names hook types needs `--hook-type pre-push`. Where `pre-push` was not installed before, `default_stages: [pre-commit]` keeps the other hooks on commit instead of running them again on push; a repository whose hooks already run on push keeps them as they are. Hooks that declare their own stages, such as `trailing-whitespace` and `end-of-file-fixer` from pre-commit-hooks, run on push as well. After the change, everyone installs the hooks once more.
- Without pre-commit, a `.git/hooks/pre-push` that runs `python .agents/review/check.py` does the same for one clone, but it checks the checked-out branch, which a plain `git push` pushes; pre-commit tells it the branch being pushed.
234 changes: 234 additions & 0 deletions .agents/review/check.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,234 @@
"""
Tells whether the AI review of a branch covers its last commit, from the record the review writes to
local/reviews/<branch>.md (see README.md next to this file).

check.py say whether the review is missing or behind the branch; always exit 0
check.py --strict the same, but exit 1 unless the branch is reviewed up to its last commit
check.py --refresh when a rebase left the reviewed changes as they were under new commits,
move the record's commit to them

As the pre-push hook it checks the branch being pushed and never blocks the push: unfinished branches
are pushed to run CI. By hand it checks the branch checked out.

Only the standard library: it runs from pre-commit in a repository of any language.
"""

from __future__ import annotations

import argparse
import codecs
import hashlib
import os
import re
import shutil
import subprocess
import sys
from dataclasses import dataclass
from pathlib import Path

RECORDS = Path("local", "reviews")
COMMIT = re.compile(r"^commit:[ \t]*([0-9a-f]{7,64})[ \t]*(?=\r?$)", re.MULTILINE)
# agents write the record in UTF-8; PowerShell 5.1 redirection writes UTF-16
BOMS = ((codecs.BOM_UTF16_LE, "utf-16"), (codecs.BOM_UTF16_BE, "utf-16"), (codecs.BOM_UTF8, "utf-8-sig"))
# the diffs are read by this script, not a person: no colour, external diff or textconv a config may turn on
PLAIN_DIFF = ("diff", "--no-color", "--no-ext-diff", "--no-textconv")
HUNK_LINES = re.compile(rb"^@@ -\d+(?:,\d+)? \+\d+(?:,\d+)? @@")
HOW = (
"run it in your agent (/review-change, $review-change; .agents/review/README.md)"
" before opening or updating the pull request"
)

REVIEWED, BEHIND, SKIPPED = "reviewed", "behind", "skipped"


@dataclass(frozen=True)
class Branch:
name: str
head: str
base: str | None
top: Path


@dataclass(frozen=True)
class Record:
path: Path
text: str
codec: str | None


def main() -> int:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument(
"--strict", action="store_true", help="exit 1 unless the branch is reviewed up to its last commit"
)
parser.add_argument(
"--refresh", action="store_true", help="move the record's commit to the same changes under new commits"
)
args = parser.parse_args()

try:
status, message = check(refresh=args.refresh)
except Exception as error: # noqa: BLE001, RUF100 - the hook must never block a push, so a broken check only says so
status, message = BEHIND, f"AI review: the check failed: {error!r}"
if status == SKIPPED and not args.strict:
return 0
sys.stdout.write(message + "\n")
return 1 if args.strict and status != REVIEWED else 0


def check(*, refresh: bool) -> tuple[str, str]:
branch, skipped = branch_to_check()
if branch is None:
return SKIPPED, skipped
relative = RECORDS / f"{branch.name}.md"
path = branch.top / relative
if not path.is_file():
return BEHIND, f"AI review of {branch.name}: not run yet; {HOW}"
record = Record(path, *read_record(path))
reviewed, missing = reviewed_commit(record.text, relative)
if reviewed is None:
return BEHIND, f"AI review of {branch.name}: {missing}; {HOW}"
if reviewed == branch.head:
return REVIEWED, f"AI review: {branch.name} is reviewed up to its last commit"
if git("merge-base", "--is-ancestor", reviewed, branch.head) is not None:
return since_review(branch, reviewed)
return after_rewrite(branch, reviewed, record, refresh=refresh)


def branch_to_check() -> tuple[Branch | None, str]:
"""The branch to check, or none and why it is skipped."""
top = git("rev-parse", "--show-toplevel")
if top is None:
return None, "AI review: not in a git work tree, or git is missing"
name, head = pushed_branch()
if name is None or head is None:
return None, "AI review: no branch to check (detached HEAD)"
base = base_ref()
if name in default_names(base):
return None, f"AI review: {name} is the default branch"
return Branch(name, head, base, Path(top)), ""


def reviewed_commit(text: str, relative: Path) -> tuple[str | None, str]:
"""The commit the record names, or none and what is wrong with the record."""
found = COMMIT.search(text)
if found is None:
return None, f"{relative.as_posix()} names no commit"
reviewed = git("rev-parse", "--verify", "--quiet", f"{found.group(1)}^{{commit}}")
if reviewed is None:
return None, f"the reviewed commit {found.group(1)[:8]} is not in this clone"
return reviewed, ""


def since_review(branch: Branch, reviewed: str) -> tuple[str, str]:
changed = branch_files_changed(reviewed, branch.head, branch.base)
if not changed:
return REVIEWED, f"AI review: {branch.name} is reviewed; nothing of its own changed since {reviewed[:8]}"
return (
BEHIND,
f"AI review of {branch.name}: {len(changed)} file(s) changed since the review at {reviewed[:8]}; {HOW}",
)


def after_rewrite(branch: Branch, reviewed: str, record: Record, *, refresh: bool) -> tuple[str, str]:
# a rebase keeps the reviewed changes under new commits: compare the changes, not the commits
if branch.base is None or not same_changes(reviewed, branch.head, branch.base):
return (
BEHIND,
f"AI review of {branch.name}: rewritten since the review at {reviewed[:8]}, with other changes; {HOW}",
)
if refresh and record.codec is not None:
record.path.write_bytes(COMMIT.sub(f"commit: {branch.head}", record.text, count=1).encode(record.codec))
return (
REVIEWED,
f"AI review: {branch.name} carries the reviewed changes; the record now names {branch.head[:8]}",
)
return REVIEWED, f"AI review: {branch.name} carries the changes reviewed at {reviewed[:8]} under new commits"


def pushed_branch() -> tuple[str | None, str | None]:
local = os.environ.get("PRE_COMMIT_LOCAL_BRANCH")
if local:
# pre-commit's pre-push: the branch being pushed and its sha, which need not be the checked-out ones
head = os.environ.get("PRE_COMMIT_TO_REF") or git("rev-parse", "--verify", "--quiet", local)
if not local.startswith("refs/heads/"):
# `git push origin HEAD` names no branch: it is the checked-out one when the shas agree
if head is None or head != git("rev-parse", "--verify", "--quiet", "HEAD"):
return None, None
local = git("symbolic-ref", "--quiet", "HEAD")
if not local:
return None, None
return local.removeprefix("refs/heads/"), head
ref = git("symbolic-ref", "--quiet", "HEAD")
if not ref:
return None, None
return ref.removeprefix("refs/heads/"), git("rev-parse", "--verify", "--quiet", ref)


def base_ref() -> str | None:
named = git("symbolic-ref", "--quiet", "--short", "refs/remotes/origin/HEAD")
candidates = [named] if named else ["origin/master", "origin/main"]
return next((ref for ref in candidates if git("rev-parse", "--verify", "--quiet", ref)), None)


def default_names(base: str | None) -> set[str]:
names = {"master", "main"}
if base:
names.add(base.split("/", 1)[-1])
return names


def read_record(path: Path) -> tuple[str, str | None]:
"""The record's text and the codec to write it back with; no codec when it is not UTF-8 or UTF-16."""
raw = path.read_bytes()
for bom, codec in BOMS:
if raw.startswith(bom):
return raw.decode(codec, errors="replace"), codec
try:
return raw.decode("utf-8"), "utf-8"
except UnicodeDecodeError:
return raw.decode("utf-8", errors="replace"), None


def same_changes(reviewed: str, head: str, base: str) -> bool:
before = changes_id(reviewed, base)
return before is not None and before == changes_id(head, base)


def changes_id(commit: str, base: str) -> str | None:
fork = git("merge-base", base, commit)
if fork is None:
return None
diff = run(*PLAIN_DIFF, fork, commit)
if not diff:
return None
# on a new base the same changes differ only in blob ids and line numbers; whitespace and context
# lines still count, so a re-indented or moved line is a change to review again
lines = [HUNK_LINES.sub(b"@@", line) for line in diff.split(b"\n") if not line.startswith(b"index ")]
return hashlib.sha256(b"\n".join(lines)).hexdigest()


def branch_files_changed(reviewed: str, head: str, base: str | None) -> set[str]:
changed = set((git(*PLAIN_DIFF, "--name-only", reviewed, head) or "").splitlines())
if base is None:
return changed
# a merge of the default branch brings other people's changes: only the branch's own files count
own = set((git(*PLAIN_DIFF, "--name-only", f"{base}...{head}") or "").splitlines())
return changed & own


def git(*args: str) -> str | None:
output = run(*args)
return None if output is None else output.decode("utf-8", errors="replace").strip()


def run(*args: str) -> bytes | None:
executable = shutil.which("git")
if not executable:
return None
result = subprocess.run([executable, *args], capture_output=True, check=False) # noqa: S603, RUF100
return result.stdout if result.returncode == 0 else None


if __name__ == "__main__":
sys.exit(main())
Loading
Loading