Skip to content

v1.1: harness fixes and test curation so the reference passes its own grader - #70

Open
john-b-yang wants to merge 14 commits into
mainfrom
v1.1
Open

john-b-yang wants to merge 14 commits into
mainfrom
v1.1

Conversation

@john-b-yang

Copy link
Copy Markdown
Contributor

Reports said that for several dozen of the 200 tasks, the reference solution failed a large share of its own tests. This PR fixes the evaluation-side causes and curates the remaining unreliable tests, so that on an ordinary Linux host with internet, gold evaluation scores 1.0 on every task, apart from occasional flakes (below).

Harness fixes (affect every submission, not just gold)

  • Pin test-time pip installs to the image's versions. Many eval/run.sh scripts run pip install -U pytest …. pytest 9.1 refuses to load the image's preinstalled libtmux plugin, so pytest died before writing results.xml (10 tasks, e.g. deadnix, dust, cppcheck, csview). After compile, the image's packages are frozen into a constraints file exported as PIP_CONSTRAINT for every test run.
  • Fix the timeout-retry plugin crashing pytest when a branch never installs pytest-timeout (jsonschema: 1101 tests lost) or runs PYTHONPATH=. pytest (i3-style, tuc).
  • Retry branches when pytest-xdist aborts with assert not crashitem, instead of silently marking the rest of the branch not run (seen in duckdb and fzf: 859 and 154 tests).
  • Unpack test archives with the container's umask. Archives store files as 0664/0775; the expected outputs were captured from a git checkout (0644/0755). Fixes permission-printing tests in walk, felix, broot, xcp and fselect.
  • Undo working-directory changes made by a test body, so a test that chdirs without restoring doesn't break later tests on the same xdist worker (miller 28, zstd 11). Directory changes made by fixtures (e.g. brotli's session fixture) are kept.
  • --build-internet: compile still runs with internet blocked by default. This flag lifts the block for trusted builds such as gold validation, whose build scripts download toolchains and dependencies. Without it, 170/200 gold builds fail on an ordinary host. Note: the DNS-blackhole block does not stop traffic through an HTTP(S) proxy that Docker injects into containers.
  • Per-task compile time limit: compile_timeout: <seconds> in a task's task.yaml (default stays 900s). jgm__pandoc gets 1800s; its reference build takes ~1000s at 10 CPUs.

Test curation (tests.json)

  • 265 filesystem-dependent size tests in 7 disk-usage tasks (dust, dutree, duc, parallel-disk-usage, dua-cli, pls, mdbook), recorded as gold_flaky: sizes depend on the grading host's filesystem.
  • 108 + 30 + 10 tests that can't pass reliably on gold: tests assuming /workspace is the original git checkout, drift in outside services (curl wording, httpbin responses, live DNS), date-dependent checks, races, and references slower than the test's timeout (gold_fail / gold_flaky / slow_or_hang).
  • 223 tests that pass on a dummy echo "hello world" executable (dummy_pass), mostly duckdb.
  • 2 ffmpeg banner tests that only fail for a from-source build.

Outside this repo

  • ffmpeg__ffmpeg.360a402 FATE samples now ship in test archive 3f244e346e7e on Hugging Face (programbench/ProgramBench-Tests commit 5954a19). They were only in task_v6, not the cleanroom grading image, so ~1030 FATE tests were unpassable for every submission. The samples are unpacked into /workspace at grading time only, never during inference.

Verification

Gold submissions were built from each task's reference build script, then evaluated with programbench eval --build-internet on an AWS host (XFS, ordinary network), all 200 tasks, two full confirmation rounds plus targeted reruns:

  • every task at 1.0 with the curated ignore lists, including pandoc with its 1800s limit;
  • a public-harness dummy run over all 200 tasks shows no remaining scored test that a trivial executable passes;
  • pytest (73 tests) and ruff (0.15.7) pass.

Known residuals: astaxie__bat still calls httpbin.org (rate limit: 2 req/s), so a few of its tests flake per run (~0.99–1.0); other rare flakes are possible. jgm__pandoc's gold build sometimes fails when cabal picks a dead Hackage mirror.

Related issues

Many generated eval/run.sh scripts `pip install -U pytest ...`, which pulls
whatever is newest at eval time. pytest 9.1 refuses to load the libtmux
plugin preinstalled in the task images ("Marks cannot be applied to
fixtures"), so pytest dies before writing results.xml and the whole branch
reports results_read_failed. This hit 10 test-set instances (deadnix, dust,
cppcheck, hush, dutree, caps-log, svd2rust, ethabi, rumdl, csview).

After compile, freeze the image's installed packages into
/etc/programbench/pip-constraints.txt and export it as PIP_CONSTRAINT for
every test run. Upgrades then resolve to the shipped versions, while
installs of packages the image lacks still work.
The injected `-p programbench_pytest_timeout` plugin (13ad325) broke pytest
startup in two cases, losing every test on the branch:

- the branch's run.sh never installs pytest-timeout, so the plugin's
  `import pytest_timeout` fails (stranger6667__jsonschema, 1101 tests).
  The plugin now no-ops without pytest-timeout: there are no timeouts to
  re-arm.
- run.sh runs `PYTHONPATH=. pytest`, dropping the plugin dir from
  PYTHONPATH (altdesktop__i3-style, riquito__tuc). The plugin dir is now
  also registered via a .pth file in site-packages.
test_multiple_log_flags_combination and
test_verbose_level_shows_additional_details assert the version banner
starts with `git-2026`. A gold submission without .git gets the harness's
synthetic 2000-01-01 commit, so a from-source build prints `git-2000-...`
and these fail deterministically (build-stamp drift, gold_fail).
pytest-xdist's controller occasionally aborts the whole session with an
INTERNALERROR (`assert not crashitem` in dsession.worker_workerfinished).
The JUnit XML then holds only the tests run so far plus one "pytest
internal" error, and every remaining test is reported not_run. Seen once
each in duckdb (859 tests lost) and fzf (154) in a full 200-instance gold
run under load; not reproducible on rerun.

count_worker_crashes only recognised "worker 'gwN' crashed", so the
existing branch retry never triggered. Count the crashitem abort as a
crash so the branch is retried and the best attempt kept.
These tests compare sizes the binary reads from the filesystem (directory
and symlink sizes, hard-link/block accounting, and totals, bar charts and
column alignment derived from them). The expected outputs were captured on
one filesystem; on a host whose Docker storage uses another (e.g. btrfs)
the reference prints different sizes and fails, as would a perfect
reimplementation. All 265 passed in two runs on the original hosts and
failed on a btrfs-backed host.

dutree 137, dust 41, duc 32, parallel-disk-usage 25, mdbook 15,
dua-cli 8, pls 7. Recorded as gold_flaky (environment-dependent).
Reference (gold) build scripts download toolchains and dependencies, so
with the compile-time DNS blackhole always on, gold-from-source fails to
compile on an ordinary host (170/200 instances). It only appeared to work
on hosts where Docker injects an HTTP(S) proxy into containers, which the
DNS blackhole does not stop.

The block stays on by default so a submission's compile.sh can't download
the original program at build time. `programbench eval --build-internet`
lifts it for trusted builds such as gold validation.
Test archives store files as 0664/0775, and tar running as root restores
those bits verbatim. The tests were generated, and their expected outputs
captured, against a git checkout, where files come out 0644/0755 (git only
tracks the exec bit). Tests that print or assert permissions therefore
failed on gold: walk, felix, broot, xcp, fselect.

Extract test archives with --no-same-permissions so the umask applies, as
with a git checkout. Submissions are unpacked unchanged.
Some generated tests chdir (often into a temp dir they then delete) and
never change back, so later tests on the same xdist worker start
elsewhere: `./executable` is not found (miller, 28 tests) or os.getcwd()
raises (zstd, 11). Which tests break depends on xdist scheduling.

The injected pytest plugin now returns each test to the directory pytest
was invoked from, which every test assumes.
From a full gold run on an ordinary host (AWS, internet on), excluding
failures fixed in the harness:

- gold_fail (59): tests that assume /workspace is the private mirror's
  git checkout (git-graph, git-trim, delta, onefetch, keifu, lazygit; 26);
  drift in outside tools/services (dirble curl wording, xh httpbin reason
  phrases, dog live DNS; 30); build differences vs golden (ctags, scc; 3).
- gold_flaky (32): live services (bat httpbin rate limit, gittype GitHub
  trending), date-dependent checks (gomplate DST, zk "today"), timing
  races (ffmpeg, sqlite, oha, peco, igrep, ast-grep).
- slow_or_hang (17): reference exceeds the test's timeout (oranda,
  errcheck, rustowl).
Resetting the cwd before every test (229f1f4) broke google__brotli, whose
session-scoped autouse fixture chdirs once for the whole run: 426 tests
lost "./executable". Instead, record the cwd before each test's call
phase and restore it afterwards if the body changed it. Leaks from test
bodies (miller, zstd) are still undone; fixture-established directories
are kept.
Failed in one full gold run on an ordinary host after passing in the
previous one: live URL fetches that got an HTML error page back (gron 10,
direnv 6), httpbin.org rate limiting (bat 6), DNS timeouts (dog 5), and
timing races (peco 1, sqlite 2). Recorded as gold_flaky.
A public-harness eval of all 200 instances against a trivial executable
(`echo "hello world"`) found 223 scored tests that pass anyway, so they
don't distinguish a real implementation from a stub: duckdb 188, fx 16,
chamber 16, solar 1, peco 1, xplr 1. Recorded as dummy_pass.
bat 6 (httpbin.org rate limiting), oranda 2 (live GitHub release links,
port check), gittype 1, solar 1. Recorded as gold_flaky.
compile.sh was capped at a hard-coded 900s. pandoc's reference build takes
~1000s at the default 10 CPUs, so its gold eval always failed to compile.

A task can now set `compile_timeout: <seconds>` in its task.yaml
(DEFAULT_COMPILE_TIMEOUT = 900 otherwise); the limit applies to every
submission for that task. jgm__pandoc gets 1800s (~1.7x the measured
build); with it, gold passes all 5214 tests.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant