Skip to content

Fix RDMA polling initialization failure cleanup - #3750

Open
yanglimingcn wants to merge 1 commit into
apache:masterfrom
yanglimingcn:bugfix/rdma_polling_initialize
Open

yanglimingcn wants to merge 1 commit into
apache:masterfrom
yanglimingcn:bugfix/rdma_polling_initialize

Conversation

@yanglimingcn

Copy link
Copy Markdown
Contributor

Reset the running state and join already-started pollers when creation fails. Release failed thread arguments, preserve the error code, and clear joined thread IDs.

Add regression coverage for creation failures, retries, and repeated release in both bthread and pthread-backed modes.

Validation: git diff --check passed. Build and unit tests were blocked by missing Protobuf development dependencies and protoc.

What problem does this PR solve?

Issue Number: resolve

Problem Summary:

What is changed and the side effects?

Changed:

Side effects:

  • Performance effects:

  • Breaking backward compatibility:


Check List:

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The Bazel RDMA unit test still references a UNIT_TEST-only hook without receiving that definition, causing an undefined-symbol link failure and leaving the injection path unavailable.

1 open issue
What changed in this PR

Fixes RDMA polling initialization failure cleanup by rolling back partially created poller groups, preserving startup errors, releasing failed thread arguments, and supporting retries and repeated release.

Changes:

  • Reset polling state, join already-started pollers, and clear thread IDs after creation failures.
  • Add RAII-based argument cleanup and error preservation during poller startup.
  • Add parameterized regression tests for failure points and bthread- and pthread-backed modes.
File Description
test/​brpc_rdma_unittest.cpp Adds lifecycle regression tests for poller creation failures, retries, repeated release, and both threading modes.
src/​brpc/​rdma/​rdma_endpoint.cpp Implements poller startup rollback, argument ownership cleanup, error preservation, test injection, and thread ID reset.

Review effort: Lite

Comment on lines +1957 to +1961
#if defined(UNIT_TEST)
// Inject poller creation failures without requiring an RDMA device.
decltype(&bthread_start_background) g_start_rdma_poller_for_test =
bthread_start_background;
#endif
Comment thread src/brpc/rdma/rdma_endpoint.cpp Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Two critical concurrency issues remain in RDMA poller initialization and rollback, allowing release or competing initialization to race with startup and leave the group stopped or pollers unjoined.

3 open issues

Review effort: Lite

Comment thread src/brpc/rdma/rdma_endpoint.cpp Outdated
Comment on lines 1974 to 1975
if (!group.initialized.compare_exchange_strong(expected, true)) {
return 0;
Comment thread src/brpc/rdma/rdma_endpoint.cpp Outdated
Comment on lines +2063 to +2069
for (auto& poller : pollers) {
if (poller.tid != INVALID_BTHREAD) {
bthread_join(poller.tid, nullptr);
poller.tid = INVALID_BTHREAD;
}
}
group.initialized.store(false, std::memory_order_release);
Comment thread src/brpc/rdma/rdma_endpoint.cpp Outdated
bool expected = false;
if (!running.compare_exchange_strong(expected, true)) {
if (!group.initialized.compare_exchange_strong(expected, true)) {
return 0;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When A fails to create the second poller and waits for the first one to exit, initialized is still true. At that point, B with the same tag will return 0 here, but A will subsequently clear the entire group. B's Channel can pass initialization, yet no poller will handle its registered CQ. Please return 0 only for groups that have started successfully; during creation or rollback, wait for the result or return a retryable error. The current assertion that initialize_during_rollback returns 0 also needs to change.

Comment thread src/brpc/rdma/rdma_endpoint.cpp Outdated
poller.tid = INVALID_BTHREAD;
}
}
group.initialized.store(false, std::memory_order_release);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If A pauses just after setting initialized to true, B can skip all invalid tid values here and clear initialized. When A resumes, it will still start the entire group of pollers, allowing the next initialization to acquire the group again and overwrite the active threads' tid values. Please use the same per-tag lifecycle protection for release, initialization, and rollback, and add a case that triggers release before the first poller is created.

Comment thread src/brpc/rdma/rdma_endpoint.cpp Outdated
running.store(false, std::memory_order_relaxed);
for (size_t j = 0; j < i; ++j) {
bthread_join(pollers[j].tid, nullptr);
pollers[j].tid = INVALID_BTHREAD;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the init_fn callbacks of two pollers need to wait for each other to become ready, and the second poller fails to be created after the first enters its callback, the join here will wait forever for the first poller to exit. running=false cannot interrupt the callback because the poller reads it only after init_fn returns. One option is to have new threads wait for the result of the entire batch creation, then execute the callbacks on success or exit directly on failure.

Reset the running state and join already-started pollers when creation fails. Release failed thread arguments, preserve the error code, and clear joined thread IDs.

Add regression coverage for creation failures, retries, and repeated release in both bthread and pthread-backed modes.

Validation: git diff --check passed. Build and unit tests were blocked by missing Protobuf development dependencies and protoc.
@yanglimingcn
yanglimingcn force-pushed the bugfix/rdma_polling_initialize branch from 558092a to e2fa894 Compare November 19, 2026 11:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants