Repository navigation
Conversation
38c8ffa to
b259261
Compare
75509ce to
481ac62
Compare
Merge the current heads of apache#2102, apache#2103, apache#2104 and apache#2105. Keep resource hooks independent of native completion reports, adopt the new VM counter layout, and remove the superseded relallfrozen catalog changes. Account for AO compaction index insertions in the database resource aggregate without attributing the all-index sample to one relation. Preserve native timing, work and interruption accounting exactly once. Update catalog/statistics format versions, regression expectations and Meson TAP registration. Cover AO/AOCO resource aggregation and preservation of native page counters across work-only resets. Validation: full assert-enabled macOS build; 14 core TAP files (210 tests), 5 existing extension TAP files (81 tests), and AO/AOCO resource TAP coverage (14 tests). macOS build uses diagnostic overrides for pre-existing warnings outside these changes. Distributed cluster regressions not run. Assisted-by: ChatGPT
Integrate PRs apache#2102, apache#2103, apache#2104 and apache#2105 on current main. Preserve the existing extension integration and the vacuum_stats expected-output whitespace fix.
481ac62 to
d8555ab
Compare
Integrate PRs apache#2102, apache#2103, apache#2104 and apache#2105 on current main. Preserve the existing extension integration and the vacuum_stats expected-output whitespace fix.
I can't download file unfortunately because it shows 'Not Found'. Can you check it please? |
23f1ec7 to
00bc308
Compare
|
|
I hit a coordinator crash with this patch (head Symptom
if (verbose)
index_start = vacrelstats->extstats->indexes; /* extstats == NULL */In gdb: Cause
No worker change or fault injection is needed: cancelling a plain VACUUM with Ctrl-C or Repro (run in one psql session) CREATE TABLE ao_big (a int, b text) USING ao_row DISTRIBUTED BY (a);
CREATE INDEX ON ao_big(a);
INSERT INTO ao_big SELECT i, repeat('x', 100) FROM generate_series(1, 3000000) i;
DELETE FROM ao_big WHERE a % 2 = 0;
-- 1. non-VERBOSE vacuum cancelled midway (Ctrl-C works too)
SET statement_timeout = '1500ms';
VACUUM ao_big;
RESET statement_timeout;
-- 2. same session, same table
VACUUM (VERBOSE) ao_big; -- coordinator: terminated by signal 11It also reproduces deterministically with Possible fixes
|
Understood. Thank you for the case, I'm investigating it and after fixing I'll run some extra scenarios to check it just in case. |
d0d789d to
49bc5dc
Compare
Thanks for the detailed reproducer. You’re right: the cleanup missed phase 0, so a VERBOSE retry could reuse statistics left by an aborted non-VERBOSE vacuum. |
Thank you for your review. I’ve addressed all the reported issues. |
Snapshot buffer activity and block I/O timing around heap, index and AO vacuum work. Assemble backend-local resource records and accumulate AO resources over active phases. Keep relation-local block counts, and subtract index resources from table-only totals to avoid double counting. Introduce optional instrumentation and its consumer hook. Allocate it only when a consumer is installed. Persistent storage and SQL access through the extension follow separately. Native timing and work counters are introduced in independent branches.
Add WAL record, full-page-image and byte deltas to the backend-local VACUUM resource measurements. Snapshot pgWalUsage around the same heap, index and AO phases as buffer usage. Keep index WAL separate from table resources and accumulate only active AO phases. Reuse PostgreSQL's existing pgWalUsage instrumentation. Delivery, persistence and SQL exposure of these counters are added by the separate extension reporting commit.
Print buffer, WAL and I/O measurements for heap work, index passes, and each AO vacuum phase. Exclude index resources from the enclosing table or phase report, include local buffer writes, and keep ordinary VACUUM quiet. Remove the extension hook declaration, definition and collection gates from the instrumentation prerequisite. The extension patch introduces that interface together with its consumer. Drop unused AO aggregate reporting from the core-only patch. Add core TAP coverage without an extension and document VERBOSE output.
Reset AO statistics at phase 0 so a VERBOSE retry after an aborted plain VACUUM cannot reuse stale state or dereference NULL extstats. Measure complete parallel index phases after worker usage is merged, then subtract those totals from the heap report.
7c12d2f to
071ca6e
Compare
|
Thanks for fixing the AO retry crash. I read the current head (071ca6e). The accounting looks right to me: serial index passes are subtracted one by one, the parallel phase is measured as a whole after the workers' usage is merged, and the AO phases subtract only the index work done during that phase. One question before I approve:
Smaller things, none blocking:
|
Thank you for the review and feedback! You’re right - I missed adding the test to CI. I’ve added it to the ic-tap job using PROVE_TESTS=t/029_vacuum_resource_verbose.pl. All 94 assertions passed including the AO retry cases. I also added explicit NULL checks for the AO statistics pointers, removed the unused OID fields and redundant comment, moved the function-timing comment back above the correct struct, and replaced the test’s copyright line with the ASF license header. |
What does this PR do?
Prepare per-operation buffer and WAL measurements for VACUUM resource
reporting.
Capture buffer activity, block I/O time, WAL records, full-page images and
WAL bytes around heap and index vacuum operations. Accumulate resource usage
across active append-optimized vacuum phases.
Keep table and index measurements separate by subtracting index resource
usage from table totals.
Type of Change
Breaking Changes
Test Plan
make installcheckmake -C src/test installcheck-cbdb-parallelChecklist