Skip to content

Add optional VACUUM buffer and WAL instrumentation - #2103

Open
Alena0704 wants to merge 6 commits into
apache:mainfrom
Alena0704:vacuum_stats_02_instrumentation
Open

Alena0704 wants to merge 6 commits into
apache:mainfrom
Alena0704:vacuum_stats_02_instrumentation

Conversation

@Alena0704

@Alena0704 Alena0704 commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

What does this PR do?

Prepare per-operation buffer and WAL measurements for VACUUM resource
reporting.

Capture buffer activity, block I/O time, WAL records, full-page images and
WAL bytes around heap and index vacuum operations. Accumulate resource usage
across active append-optimized vacuum phases.

Keep table and index measurements separate by subtracting index resource
usage from table totals.

Type of Change

  • Bug fix (non-breaking change)
  • New feature (non-breaking change)
  • Breaking change (fix or feature with breaking changes)
  • Documentation update

Breaking Changes

Test Plan

  • Unit tests added/updated
  • Integration tests added/updated
  • Passed make installcheck
  • Passed make -C src/test installcheck-cbdb-parallel

Checklist

@Alena0704
Alena0704 force-pushed the vacuum_stats_02_instrumentation branch from 38c8ffa to b259261 Compare October 9, 2026 16:29
@Alena0704
Alena0704 force-pushed the vacuum_stats_02_instrumentation branch from 75509ce to 481ac62 Compare November 1, 2026 14:34
@Alena0704
Alena0704 marked this pull request as ready for review November 1, 2026 14:34
Alena0704 added a commit to Alena0704/cloudberry that referenced this pull request Nov 1, 2026
Merge the current heads of apache#2102, apache#2103, apache#2104 and apache#2105. Keep resource
hooks independent of native completion reports, adopt the new VM counter
layout, and remove the superseded relallfrozen catalog changes.

Account for AO compaction index insertions in the database resource
aggregate without attributing the all-index sample to one relation.
Preserve native timing, work and interruption accounting exactly once.
Update catalog/statistics format versions, regression expectations and
Meson TAP registration. Cover AO/AOCO resource aggregation and preservation
of native page counters across work-only resets.

Validation: full assert-enabled macOS build; 14 core TAP files (210 tests),
5 existing extension TAP files (81 tests), and AO/AOCO resource TAP
coverage (14 tests). macOS build uses diagnostic overrides for pre-existing
warnings outside these changes. Distributed cluster regressions not run.

Assisted-by: ChatGPT
@Alena0704
Alena0704 removed the request for review from chenjinbao1989 November 2, 2026 18:18
Alena0704 added a commit to Alena0704/cloudberry that referenced this pull request Nov 5, 2026
Integrate PRs apache#2102, apache#2103, apache#2104 and apache#2105 on current main.
Preserve the existing extension integration and the vacuum_stats
expected-output whitespace fix.
@Alena0704
Alena0704 force-pushed the vacuum_stats_02_instrumentation branch from 481ac62 to d8555ab Compare November 5, 2026 09:27
Alena0704 added a commit to Alena0704/cloudberry that referenced this pull request Nov 5, 2026
Integrate PRs apache#2102, apache#2103, apache#2104 and apache#2105 on current main.
Preserve the existing extension integration and the vacuum_stats
expected-output whitespace fix.

@leborchuk leborchuk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM,

Some issues was found by OCR, but they are minor,

cloudberry-pr2103-ocr.txt

@Alena0704

Copy link
Copy Markdown
Collaborator Author

LGTM,

Some issues was found by OCR, but they are minor,

cloudberry-pr2103-ocr.txt

I can't download file unfortunately because it shows 'Not Found'. Can you check it please?

@Alena0704
Alena0704 force-pushed the vacuum_stats_02_instrumentation branch 2 times, most recently from 23f1ec7 to 00bc308 Compare November 10, 2026 11:00
@leborchuk

Copy link
Copy Markdown
Contributor

LGTM,
Some issues was found by OCR, but they are minor,
cloudberry-pr2103-ocr.txt

I can't download file unfortunately because it shows 'Not Found'. Can you check it please?


─── src/backend/commands/vacuumparallel.c:886-887 ───
[bug · medium] Parallel index reports are logged and discarded without contributing to
`vacrel->extVacReportIdx`. Both leader-only and parallel-safe index passes use this function when
parallel vacuum is active, so neither reaches `accumulate_idxs_vacuum_statistics()`. Moreover,
`parallel_vacuum_process_all_indexes()` merges worker buffer/WAL usage into the leader through
`InstrAccumParallelQuery()`. Consequently the final report labeled "heap (excluding indexes)"
includes both leader and worker index work, duplicating the separately logged index usage. Propagate
the parallel-pass totals back to the heap accumulator (or measure and subtract the complete parallel
phase after worker aggregation), and cover this with a parallel-vacuum accounting test.



─── src/test/modules/test_misc/t/029_vacuum_resource_verbose.pl:49-49 ───
[test · medium] The report terminator does not match psql's actual diagnostic prefixes.
Cluster::psql runs `psql -f -`, so subsequent messages start with `psql:<stdin>:<line>: INFO:`, not
`INFO:`. Consequently each phase capture can include all subsequent reports, allowing the
nonnegative-buffer, WAL, and timing assertions to pass on another phase's counters even when this
phase is wrong. Bound the capture to the next resource-report header or recognize psql's diagnostic
prefix so these assertions validate the requested phase only.

@my-ship-it

Copy link
Copy Markdown
Contributor

I hit a coordinator crash with this patch (head 00bc3083dc6, cassert build, 3-segment demo cluster).

Symptom

VACUUM (VERBOSE) on an AO table segfaults the coordinator backend at vacuum_ao.c:483, and the postmaster then reinitializes the whole cluster:

	if (verbose)
		index_start = vacrelstats->extstats->indexes;   /* extstats == NULL */

In gdb: vacuum_rel -> table_relation_vacuum -> ao_vacuum_rel, ao_vacuum_phase = 0, *vacrelstats = {relid = 17029, extstats = 0x0}.

Cause

vacrelstats is function-static, and extstats is allocated only when the vacuum that created it was VERBOSE (init_vacrelstats(verbose)). If a non-VERBOSE AO vacuum fails or is cancelled partway, it leaves vacrelstats behind with extstats == NULL. The new leftover cleanup drops it only when the phase is PRE_CLEANUP or the relid differs. But the coordinator first calls ao_vacuum_rel with phase 0, so a following VACUUM (VERBOSE) on the same table in the same session reuses the leftover. Line 483 then dereferences NULL.

No worker change or fault injection is needed: cancelling a plain VACUUM with Ctrl-C or statement_timeout, then running VERBOSE, is enough.

Repro (run in one psql session)

CREATE TABLE ao_big (a int, b text) USING ao_row DISTRIBUTED BY (a);
CREATE INDEX ON ao_big(a);
INSERT INTO ao_big SELECT i, repeat('x', 100) FROM generate_series(1, 3000000) i;
DELETE FROM ao_big WHERE a % 2 = 0;

-- 1. non-VERBOSE vacuum cancelled midway (Ctrl-C works too)
SET statement_timeout = '1500ms';
VACUUM ao_big;
RESET statement_timeout;

-- 2. same session, same table
VACUUM (VERBOSE) ao_big;   -- coordinator: terminated by signal 11

It also reproduces deterministically with gp_inject_fault('vacuum_ao_after_compact', 'error', <coordinator dbid>) in step 1 instead of the timeout.

Possible fixes

  • Guard on the allocation rather than the option: if (vacrelstats->extstats != NULL), and the same for the other extstats uses (ao_measure_index_resources, the phase report). Or allocate extstats lazily when verbose is set and it is still NULL.
  • More generally, a leftover from a failed vacuum is still reused whenever the next call is not PRE_CLEANUP and has the same relid. That includes phase 0 on the QD, so stale counters and starttime can carry over even without VERBOSE. It may be cleaner to also reset at phase 0, or to free vacrelstats on error (e.g. reset it at transaction abort) instead of detecting leftovers by relid.

@Alena0704

Copy link
Copy Markdown
Collaborator Author

I hit a coordinator crash with this patch (head 00bc3083dc6, cassert build, 3-segment demo cluster).

Symptom

VACUUM (VERBOSE) on an AO table segfaults the coordinator backend at vacuum_ao.c:483, and the postmaster then reinitializes the whole cluster:

	if (verbose)
		index_start = vacrelstats->extstats->indexes;   /* extstats == NULL */

In gdb: vacuum_rel -> table_relation_vacuum -> ao_vacuum_rel, ao_vacuum_phase = 0, *vacrelstats = {relid = 17029, extstats = 0x0}.

Cause

vacrelstats is function-static, and extstats is allocated only when the vacuum that created it was VERBOSE (init_vacrelstats(verbose)). If a non-VERBOSE AO vacuum fails or is cancelled partway, it leaves vacrelstats behind with extstats == NULL. The new leftover cleanup drops it only when the phase is PRE_CLEANUP or the relid differs. But the coordinator first calls ao_vacuum_rel with phase 0, so a following VACUUM (VERBOSE) on the same table in the same session reuses the leftover. Line 483 then dereferences NULL.

No worker change or fault injection is needed: cancelling a plain VACUUM with Ctrl-C or statement_timeout, then running VERBOSE, is enough.

Repro (run in one psql session)

CREATE TABLE ao_big (a int, b text) USING ao_row DISTRIBUTED BY (a);
CREATE INDEX ON ao_big(a);
INSERT INTO ao_big SELECT i, repeat('x', 100) FROM generate_series(1, 3000000) i;
DELETE FROM ao_big WHERE a % 2 = 0;

-- 1. non-VERBOSE vacuum cancelled midway (Ctrl-C works too)
SET statement_timeout = '1500ms';
VACUUM ao_big;
RESET statement_timeout;

-- 2. same session, same table
VACUUM (VERBOSE) ao_big;   -- coordinator: terminated by signal 11

It also reproduces deterministically with gp_inject_fault('vacuum_ao_after_compact', 'error', <coordinator dbid>) in step 1 instead of the timeout.

Possible fixes

  • Guard on the allocation rather than the option: if (vacrelstats->extstats != NULL), and the same for the other extstats uses (ao_measure_index_resources, the phase report). Or allocate extstats lazily when verbose is set and it is still NULL.
  • More generally, a leftover from a failed vacuum is still reused whenever the next call is not PRE_CLEANUP and has the same relid. That includes phase 0 on the QD, so stale counters and starttime can carry over even without VERBOSE. It may be cleaner to also reset at phase 0, or to free vacrelstats on error (e.g. reset it at transaction abort) instead of detecting leftovers by relid.

Understood. Thank you for the case, I'm investigating it and after fixing I'll run some extra scenarios to check it just in case.

@Alena0704
Alena0704 force-pushed the vacuum_stats_02_instrumentation branch 2 times, most recently from d0d789d to 49bc5dc Compare November 16, 2026 19:16
@Alena0704

Copy link
Copy Markdown
Collaborator Author

I hit a coordinator crash with this patch (head 00bc3083dc6, cassert build, 3-segment demo cluster).

Symptom

VACUUM (VERBOSE) on an AO table segfaults the coordinator backend at vacuum_ao.c:483, and the postmaster then reinitializes the whole cluster:

	if (verbose)
		index_start = vacrelstats->extstats->indexes;   /* extstats == NULL */

In gdb: vacuum_rel -> table_relation_vacuum -> ao_vacuum_rel, ao_vacuum_phase = 0, *vacrelstats = {relid = 17029, extstats = 0x0}.

Cause

vacrelstats is function-static, and extstats is allocated only when the vacuum that created it was VERBOSE (init_vacrelstats(verbose)). If a non-VERBOSE AO vacuum fails or is cancelled partway, it leaves vacrelstats behind with extstats == NULL. The new leftover cleanup drops it only when the phase is PRE_CLEANUP or the relid differs. But the coordinator first calls ao_vacuum_rel with phase 0, so a following VACUUM (VERBOSE) on the same table in the same session reuses the leftover. Line 483 then dereferences NULL.

No worker change or fault injection is needed: cancelling a plain VACUUM with Ctrl-C or statement_timeout, then running VERBOSE, is enough.

Repro (run in one psql session)

CREATE TABLE ao_big (a int, b text) USING ao_row DISTRIBUTED BY (a);
CREATE INDEX ON ao_big(a);
INSERT INTO ao_big SELECT i, repeat('x', 100) FROM generate_series(1, 3000000) i;
DELETE FROM ao_big WHERE a % 2 = 0;

-- 1. non-VERBOSE vacuum cancelled midway (Ctrl-C works too)
SET statement_timeout = '1500ms';
VACUUM ao_big;
RESET statement_timeout;

-- 2. same session, same table
VACUUM (VERBOSE) ao_big;   -- coordinator: terminated by signal 11

It also reproduces deterministically with gp_inject_fault('vacuum_ao_after_compact', 'error', <coordinator dbid>) in step 1 instead of the timeout.

Possible fixes

  • Guard on the allocation rather than the option: if (vacrelstats->extstats != NULL), and the same for the other extstats uses (ao_measure_index_resources, the phase report). Or allocate extstats lazily when verbose is set and it is still NULL.
  • More generally, a leftover from a failed vacuum is still reused whenever the next call is not PRE_CLEANUP and has the same relid. That includes phase 0 on the QD, so stale counters and starttime can carry over even without VERBOSE. It may be cleaner to also reset at phase 0, or to free vacrelstats on error (e.g. reset it at transaction abort) instead of detecting leftovers by relid.

Thanks for the detailed reproducer. You’re right: the cleanup missed phase 0, so a VERBOSE retry could reuse statistics left by an aborted non-VERBOSE vacuum.
I’ve fixed this by resetting vacrelstats at phase 0 as well as pre-cleanup. This discards stale counters and restarts progress reporting with the correct instrumentation allocation.
I also added regression coverage for AO row and column tables: an expression-index error aborts compaction, followed by a VERBOSE retry on the same relation in the same session. The test reproduces SIGSEGV before the fix and passes afterward, verifying that the backend survives and all three phase reports are present.

@Alena0704

Copy link
Copy Markdown
Collaborator Author

LGTM,
Some issues was found by OCR, but they are minor,
cloudberry-pr2103-ocr.txt

I can't download file unfortunately because it shows 'Not Found'. Can you check it please?


─── src/backend/commands/vacuumparallel.c:886-887 ───
[bug · medium] Parallel index reports are logged and discarded without contributing to
`vacrel->extVacReportIdx`. Both leader-only and parallel-safe index passes use this function when
parallel vacuum is active, so neither reaches `accumulate_idxs_vacuum_statistics()`. Moreover,
`parallel_vacuum_process_all_indexes()` merges worker buffer/WAL usage into the leader through
`InstrAccumParallelQuery()`. Consequently the final report labeled "heap (excluding indexes)"
includes both leader and worker index work, duplicating the separately logged index usage. Propagate
the parallel-pass totals back to the heap accumulator (or measure and subtract the complete parallel
phase after worker aggregation), and cover this with a parallel-vacuum accounting test.



─── src/test/modules/test_misc/t/029_vacuum_resource_verbose.pl:49-49 ───
[test · medium] The report terminator does not match psql's actual diagnostic prefixes.
Cluster::psql runs `psql -f -`, so subsequent messages start with `psql:<stdin>:<line>: INFO:`, not
`INFO:`. Consequently each phase capture can include all subsequent reports, allowing the
nonnegative-buffer, WAL, and timing assertions to pass on another phase's counters even when this
phase is wrong. Bound the capture to the next resource-report header or recognize psql's diagnostic
prefix so these assertions validate the requested phase only.

Thank you for your review. I’ve addressed all the reported issues.

Snapshot buffer activity and block I/O timing around heap, index and AO
vacuum work. Assemble backend-local resource records and accumulate AO
resources over active phases. Keep relation-local block counts, and
subtract index resources from table-only totals to avoid double counting.

Introduce optional instrumentation and its consumer hook. Allocate it only
when a consumer is installed. Persistent storage and SQL access through
the extension follow separately. Native timing and work counters are
introduced in independent branches.
Add WAL record, full-page-image and byte deltas to the backend-local
VACUUM resource measurements. Snapshot pgWalUsage around the same heap,
index and AO phases as buffer usage. Keep index WAL separate from table
resources and accumulate only active AO phases.

Reuse PostgreSQL's existing pgWalUsage instrumentation. Delivery,
persistence and SQL exposure of these counters are added by the separate
extension reporting commit.
Print buffer, WAL and I/O measurements for heap work, index passes, and
each AO vacuum phase. Exclude index resources from the enclosing table
or phase report, include local buffer writes, and keep ordinary VACUUM
quiet.

Remove the extension hook declaration, definition and collection gates
from the instrumentation prerequisite. The extension patch introduces
that interface together with its consumer. Drop unused AO aggregate
reporting from the core-only patch. Add core TAP coverage without an
extension and document VERBOSE output.
Reset AO statistics at phase 0 so a VERBOSE retry after an aborted
plain VACUUM cannot reuse stale state or dereference NULL extstats.

Measure complete parallel index phases after worker usage is merged,
then subtract those totals from the heap report.
@Alena0704
Alena0704 force-pushed the vacuum_stats_02_instrumentation branch from 7c12d2f to 071ca6e Compare November 17, 2026 14:47
@MisterRaindrop

Copy link
Copy Markdown
Contributor

Thanks for fixing the AO retry crash. I read the current head (071ca6e). The accounting looks right to me: serial index passes are subtracted one by one, the parallel phase is measured as a whole after the workers' usage is merged, and the AO phases subtract only the index work done during that phase.

One question before I approve:

  1. The new src/test/modules/test_misc/t/029_vacuum_resource_verbose.pl does not run in CI. No job runs src/test/modules/test_misc, so the green CI says nothing about this test, including the retry case for the crash. Did it pass for you on Cloudberry? Could you add src/test/modules/test_misc to the ic-tap job, or move the checks to a test that CI runs? If the older test_misc tests fail on Cloudberry, running only this file with PROVE_TESTS=t/029_vacuum_resource_verbose.pl would also work.

Smaller things, none blocking:

  1. ao_vacuum_rel() reads vacrelstats->extstats->indexes when verbose is set. The reset at phase 0 and pre-cleanup makes this safe now, but checking vacrelstats->extstats != NULL instead would not depend on that reset.
  2. reloid and indoid in LVRelState are set but never read in this PR. The comment "We need to accumulate index statistics for later subtraction from heap stats." after extVacReportIdx describes no field.
  3. In pgstat.h the new types are inserted between the comment "Working state needed to accumulate per-function-call timing statistics." and PgStat_FunctionCallUsage, so that comment now sits above the wrong struct.
  4. Cloudberry builds with make, so the meson.build change has no effect. The new test file carries the PostgreSQL copyright line; new Cloudberry files usually carry the ASF header.

@Alena0704

Copy link
Copy Markdown
Collaborator Author

Thanks for fixing the AO retry crash. I read the current head (071ca6e). The accounting looks right to me: serial index passes are subtracted one by one, the parallel phase is measured as a whole after the workers' usage is merged, and the AO phases subtract only the index work done during that phase.

One question before I approve:

  1. The new src/test/modules/test_misc/t/029_vacuum_resource_verbose.pl does not run in CI. No job runs src/test/modules/test_misc, so the green CI says nothing about this test, including the retry case for the crash. Did it pass for you on Cloudberry? Could you add src/test/modules/test_misc to the ic-tap job, or move the checks to a test that CI runs? If the older test_misc tests fail on Cloudberry, running only this file with PROVE_TESTS=t/029_vacuum_resource_verbose.pl would also work.

Smaller things, none blocking:

  1. ao_vacuum_rel() reads vacrelstats->extstats->indexes when verbose is set. The reset at phase 0 and pre-cleanup makes this safe now, but checking vacrelstats->extstats != NULL instead would not depend on that reset.
  2. reloid and indoid in LVRelState are set but never read in this PR. The comment "We need to accumulate index statistics for later subtraction from heap stats." after extVacReportIdx describes no field.
  3. In pgstat.h the new types are inserted between the comment "Working state needed to accumulate per-function-call timing statistics." and PgStat_FunctionCallUsage, so that comment now sits above the wrong struct.
  4. Cloudberry builds with make, so the meson.build change has no effect. The new test file carries the PostgreSQL copyright line; new Cloudberry files usually carry the ASF header.

Thank you for the review and feedback! You’re right - I missed adding the test to CI. I’ve added it to the ic-tap job using PROVE_TESTS=t/029_vacuum_resource_verbose.pl. All 94 assertions passed including the AO retry cases.

I also added explicit NULL checks for the AO statistics pointers, removed the unused OID fields and redundant comment, moved the function-timing comment back above the correct struct, and replaced the test’s copyright line with the ASF license header.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants