Repository navigation
v1.1: harness fixes and test curation so the reference passes its own grader - #70
Open
john-b-yang wants to merge 14 commits into
Open
john-b-yang wants to merge 14 commits into
john-b-yang wants to merge 14 commits into
Conversation
Many generated eval/run.sh scripts `pip install -U pytest ...`, which pulls
whatever is newest at eval time. pytest 9.1 refuses to load the libtmux
plugin preinstalled in the task images ("Marks cannot be applied to
fixtures"), so pytest dies before writing results.xml and the whole branch
reports results_read_failed. This hit 10 test-set instances (deadnix, dust,
cppcheck, hush, dutree, caps-log, svd2rust, ethabi, rumdl, csview).
After compile, freeze the image's installed packages into
/etc/programbench/pip-constraints.txt and export it as PIP_CONSTRAINT for
every test run. Upgrades then resolve to the shipped versions, while
installs of packages the image lacks still work.
The injected `-p programbench_pytest_timeout` plugin (13ad325) broke pytest startup in two cases, losing every test on the branch: - the branch's run.sh never installs pytest-timeout, so the plugin's `import pytest_timeout` fails (stranger6667__jsonschema, 1101 tests). The plugin now no-ops without pytest-timeout: there are no timeouts to re-arm. - run.sh runs `PYTHONPATH=. pytest`, dropping the plugin dir from PYTHONPATH (altdesktop__i3-style, riquito__tuc). The plugin dir is now also registered via a .pth file in site-packages.
test_multiple_log_flags_combination and test_verbose_level_shows_additional_details assert the version banner starts with `git-2026`. A gold submission without .git gets the harness's synthetic 2000-01-01 commit, so a from-source build prints `git-2000-...` and these fail deterministically (build-stamp drift, gold_fail).
pytest-xdist's controller occasionally aborts the whole session with an INTERNALERROR (`assert not crashitem` in dsession.worker_workerfinished). The JUnit XML then holds only the tests run so far plus one "pytest internal" error, and every remaining test is reported not_run. Seen once each in duckdb (859 tests lost) and fzf (154) in a full 200-instance gold run under load; not reproducible on rerun. count_worker_crashes only recognised "worker 'gwN' crashed", so the existing branch retry never triggered. Count the crashitem abort as a crash so the branch is retried and the best attempt kept.
These tests compare sizes the binary reads from the filesystem (directory and symlink sizes, hard-link/block accounting, and totals, bar charts and column alignment derived from them). The expected outputs were captured on one filesystem; on a host whose Docker storage uses another (e.g. btrfs) the reference prints different sizes and fails, as would a perfect reimplementation. All 265 passed in two runs on the original hosts and failed on a btrfs-backed host. dutree 137, dust 41, duc 32, parallel-disk-usage 25, mdbook 15, dua-cli 8, pls 7. Recorded as gold_flaky (environment-dependent).
Reference (gold) build scripts download toolchains and dependencies, so with the compile-time DNS blackhole always on, gold-from-source fails to compile on an ordinary host (170/200 instances). It only appeared to work on hosts where Docker injects an HTTP(S) proxy into containers, which the DNS blackhole does not stop. The block stays on by default so a submission's compile.sh can't download the original program at build time. `programbench eval --build-internet` lifts it for trusted builds such as gold validation.
Test archives store files as 0664/0775, and tar running as root restores those bits verbatim. The tests were generated, and their expected outputs captured, against a git checkout, where files come out 0644/0755 (git only tracks the exec bit). Tests that print or assert permissions therefore failed on gold: walk, felix, broot, xcp, fselect. Extract test archives with --no-same-permissions so the umask applies, as with a git checkout. Submissions are unpacked unchanged.
Some generated tests chdir (often into a temp dir they then delete) and never change back, so later tests on the same xdist worker start elsewhere: `./executable` is not found (miller, 28 tests) or os.getcwd() raises (zstd, 11). Which tests break depends on xdist scheduling. The injected pytest plugin now returns each test to the directory pytest was invoked from, which every test assumes.
From a full gold run on an ordinary host (AWS, internet on), excluding failures fixed in the harness: - gold_fail (59): tests that assume /workspace is the private mirror's git checkout (git-graph, git-trim, delta, onefetch, keifu, lazygit; 26); drift in outside tools/services (dirble curl wording, xh httpbin reason phrases, dog live DNS; 30); build differences vs golden (ctags, scc; 3). - gold_flaky (32): live services (bat httpbin rate limit, gittype GitHub trending), date-dependent checks (gomplate DST, zk "today"), timing races (ffmpeg, sqlite, oha, peco, igrep, ast-grep). - slow_or_hang (17): reference exceeds the test's timeout (oranda, errcheck, rustowl).
Resetting the cwd before every test (229f1f4) broke google__brotli, whose session-scoped autouse fixture chdirs once for the whole run: 426 tests lost "./executable". Instead, record the cwd before each test's call phase and restore it afterwards if the body changed it. Leaks from test bodies (miller, zstd) are still undone; fixture-established directories are kept.
Failed in one full gold run on an ordinary host after passing in the previous one: live URL fetches that got an HTML error page back (gron 10, direnv 6), httpbin.org rate limiting (bat 6), DNS timeouts (dog 5), and timing races (peco 1, sqlite 2). Recorded as gold_flaky.
A public-harness eval of all 200 instances against a trivial executable (`echo "hello world"`) found 223 scored tests that pass anyway, so they don't distinguish a real implementation from a stub: duckdb 188, fx 16, chamber 16, solar 1, peco 1, xplr 1. Recorded as dummy_pass.
bat 6 (httpbin.org rate limiting), oranda 2 (live GitHub release links, port check), gittype 1, solar 1. Recorded as gold_flaky.
compile.sh was capped at a hard-coded 900s. pandoc's reference build takes ~1000s at the default 10 CPUs, so its gold eval always failed to compile. A task can now set `compile_timeout: <seconds>` in its task.yaml (DEFAULT_COMPILE_TIMEOUT = 900 otherwise); the limit applies to every submission for that task. jgm__pandoc gets 1800s (~1.7x the measured build); with it, gold passes all 5214 tests.
This was referenced Nov 11, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reports said that for several dozen of the 200 tasks, the reference solution failed a large share of its own tests. This PR fixes the evaluation-side causes and curates the remaining unreliable tests, so that on an ordinary Linux host with internet, gold evaluation scores 1.0 on every task, apart from occasional flakes (below).
Harness fixes (affect every submission, not just gold)
eval/run.shscripts runpip install -U pytest …. pytest 9.1 refuses to load the image's preinstalledlibtmuxplugin, so pytest died before writingresults.xml(10 tasks, e.g. deadnix, dust, cppcheck, csview). After compile, the image's packages are frozen into a constraints file exported asPIP_CONSTRAINTfor every test run.pytest-timeout(jsonschema: 1101 tests lost) or runsPYTHONPATH=. pytest(i3-style, tuc).assert not crashitem, instead of silently marking the rest of the branch not run (seen in duckdb and fzf: 859 and 154 tests).chdirs without restoring doesn't break later tests on the same xdist worker (miller 28, zstd 11). Directory changes made by fixtures (e.g. brotli's session fixture) are kept.--build-internet: compile still runs with internet blocked by default. This flag lifts the block for trusted builds such as gold validation, whose build scripts download toolchains and dependencies. Without it, 170/200 gold builds fail on an ordinary host. Note: the DNS-blackhole block does not stop traffic through an HTTP(S) proxy that Docker injects into containers.compile_timeout: <seconds>in a task'stask.yaml(default stays 900s).jgm__pandocgets 1800s; its reference build takes ~1000s at 10 CPUs.Test curation (
tests.json)gold_flaky: sizes depend on the grading host's filesystem./workspaceis the original git checkout, drift in outside services (curl wording, httpbin responses, live DNS), date-dependent checks, races, and references slower than the test's timeout (gold_fail/gold_flaky/slow_or_hang).echo "hello world"executable (dummy_pass), mostly duckdb.Outside this repo
ffmpeg__ffmpeg.360a402FATE samples now ship in test archive3f244e346e7eon Hugging Face (programbench/ProgramBench-Testscommit 5954a19). They were only intask_v6, not the cleanroom grading image, so ~1030 FATE tests were unpassable for every submission. The samples are unpacked into/workspaceat grading time only, never during inference.Verification
Gold submissions were built from each task's reference build script, then evaluated with
programbench eval --build-interneton an AWS host (XFS, ordinary network), all 200 tasks, two full confirmation rounds plus targeted reruns:pytest(73 tests) andruff(0.15.7) pass.Known residuals:
astaxie__batstill calls httpbin.org (rate limit: 2 req/s), so a few of its tests flake per run (~0.99–1.0); other rare flakes are possible.jgm__pandoc's gold build sometimes fails when cabal picks a dead Hackage mirror.Related issues
results_read_failedbranch is fixed here (pytest pin). Theeval.-prefix name-matching issue is not addressed in this PR.mainby pinningpytest-rerunfailures==16.4; the frozen constraints here keep that pin during test runs too.