Repository navigation
Fix RDMA polling initialization failure cleanup - #3750
yanglimingcn wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The Bazel RDMA unit test still references a UNIT_TEST-only hook without receiving that definition, causing an undefined-symbol link failure and leaving the injection path unavailable.
1 open issue
What changed in this PR
Fixes RDMA polling initialization failure cleanup by rolling back partially created poller groups, preserving startup errors, releasing failed thread arguments, and supporting retries and repeated release.
Changes:
- Reset polling state, join already-started pollers, and clear thread IDs after creation failures.
- Add RAII-based argument cleanup and error preservation during poller startup.
- Add parameterized regression tests for failure points and bthread- and pthread-backed modes.
| File | Description |
|---|---|
| test/brpc_rdma_unittest.cpp | Adds lifecycle regression tests for poller creation failures, retries, repeated release, and both threading modes. |
| src/brpc/rdma/rdma_endpoint.cpp | Implements poller startup rollback, argument ownership cleanup, error preservation, test injection, and thread ID reset. |
Review effort: Lite
| #if defined(UNIT_TEST) | ||
| // Inject poller creation failures without requiring an RDMA device. | ||
| decltype(&bthread_start_background) g_start_rdma_poller_for_test = | ||
| bthread_start_background; | ||
| #endif |
96d36d4 to
558092a
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Two critical concurrency issues remain in RDMA poller initialization and rollback, allowing release or competing initialization to race with startup and leave the group stopped or pollers unjoined.
3 open issues
Review effort: Lite
| if (!group.initialized.compare_exchange_strong(expected, true)) { | ||
| return 0; |
| for (auto& poller : pollers) { | ||
| if (poller.tid != INVALID_BTHREAD) { | ||
| bthread_join(poller.tid, nullptr); | ||
| poller.tid = INVALID_BTHREAD; | ||
| } | ||
| } | ||
| group.initialized.store(false, std::memory_order_release); |
| bool expected = false; | ||
| if (!running.compare_exchange_strong(expected, true)) { | ||
| if (!group.initialized.compare_exchange_strong(expected, true)) { | ||
| return 0; |
There was a problem hiding this comment.
When A fails to create the second poller and waits for the first one to exit, initialized is still true. At that point, B with the same tag will return 0 here, but A will subsequently clear the entire group. B's Channel can pass initialization, yet no poller will handle its registered CQ. Please return 0 only for groups that have started successfully; during creation or rollback, wait for the result or return a retryable error. The current assertion that initialize_during_rollback returns 0 also needs to change.
| poller.tid = INVALID_BTHREAD; | ||
| } | ||
| } | ||
| group.initialized.store(false, std::memory_order_release); |
There was a problem hiding this comment.
If A pauses just after setting initialized to true, B can skip all invalid tid values here and clear initialized. When A resumes, it will still start the entire group of pollers, allowing the next initialization to acquire the group again and overwrite the active threads' tid values. Please use the same per-tag lifecycle protection for release, initialization, and rollback, and add a case that triggers release before the first poller is created.
| running.store(false, std::memory_order_relaxed); | ||
| for (size_t j = 0; j < i; ++j) { | ||
| bthread_join(pollers[j].tid, nullptr); | ||
| pollers[j].tid = INVALID_BTHREAD; |
There was a problem hiding this comment.
If the init_fn callbacks of two pollers need to wait for each other to become ready, and the second poller fails to be created after the first enters its callback, the join here will wait forever for the first poller to exit. running=false cannot interrupt the callback because the poller reads it only after init_fn returns. One option is to have new threads wait for the result of the entire batch creation, then execute the callbacks on success or exit directly on failure.
Reset the running state and join already-started pollers when creation fails. Release failed thread arguments, preserve the error code, and clear joined thread IDs. Add regression coverage for creation failures, retries, and repeated release in both bthread and pthread-backed modes. Validation: git diff --check passed. Build and unit tests were blocked by missing Protobuf development dependencies and protoc.
558092a to
e2fa894
Compare

Reset the running state and join already-started pollers when creation fails. Release failed thread arguments, preserve the error code, and clear joined thread IDs.
Add regression coverage for creation failures, retries, and repeated release in both bthread and pthread-backed modes.
Validation: git diff --check passed. Build and unit tests were blocked by missing Protobuf development dependencies and protoc.
What problem does this PR solve?
Issue Number: resolve
Problem Summary:
What is changed and the side effects?
Changed:
Side effects:
Performance effects:
Breaking backward compatibility:
Check List: