Repository navigation
perf(butil): replace ARM64 legacy inline assembly atomics with __atomic builtins - #3752
jimmystewpot wants to merge 4 commits into
Conversation
…ntion Signed-off-by: James Lamb <[email protected]>
…ic builtins Signed-off-by: James Lamb <[email protected]>
… hdrs Signed-off-by: James Lamb <[email protected]>
There was a problem hiding this comment.
🔵 Needs a closer look
Approval is deferred because the ARM64-specific atomic and Apple Silicon changes require native cross-platform builds, generated-instruction checks, and concurrency/runtime validation unavailable in this environment.
No open issues 🎉
What changed in this PR
This pull request modernizes ARM64 atomic operations in butil by replacing legacy inline assembly and full barriers with compiler-supported __atomic builtins, enabling LSE and outline-atomic code generation while preserving the existing API.
Changes:
- Replaced 32-bit and 64-bit ARM64 atomic implementations with explicit-memory-order
__atomicoperations. - Routed ARM64, including Apple Silicon, builds to the modern implementation and exported the header through Bazel.
- Expanded atomic operation and contention tests, with acquire/release visibility coverage still needing improvement.
| File | Description |
|---|---|
| test/atomicops_unittest.cc | Adds coverage for 64-bit atomics, CAS success and failure paths, barrier increments, and multithreaded contention. |
| src/butil/atomicops.h | Prioritizes the ARM64 GCC/Clang implementation over the legacy macOS implementation for ARM64 builds. |
| src/butil/atomicops_internals_arm64_gcc.h | Implements ARM64 atomic operations with GCC/Clang __atomic builtins and explicit memory ordering. |
| BUILD.bazel | Exports the ARM64 atomic header for the macOS Bazel target. |
Review effort: Lite
…sibility tests Signed-off-by: James Lamb <[email protected]>
I have added instructions/an expandable section in the above PR description that provides some validation of correctness, etc., for this. |
What problem does this PR solve?
Issue Number: #3753
Problem Summary:
This issue was discovered while profiling StarRocks (which relies heavily on
brpc/bthread/butilfor RPC and concurrency) on ARM64 servers, where CPU profiles showed significant time and cache-coherency stalls spent inside legacyldxr/stxrloops and unconditionaldmb ishmemory barriers.src/butil/atomicops_internals_arm64_gcc.hhas not been updated since its original 2014 Chromium import and suffers from three key limitations on modern ARM64 (__aarch64__) platforms:NoBarrier_CompareAndSwap,NoBarrier_AtomicExchange, andNoBarrier_AtomicIncrement(both 32-bit and 64-bit) use literalldxr/stxrload-linked/store-conditional assembly loops. Because these are opaque assembly strings to the compiler, neither GCC nor Clang can emit ARMv8.1-A Large System Extensions (LSE) instructions (cas,swp,ldadd,stadd) when targeting-march=armv8-a+lse/-march=armv8.2-a+, nor can they emit runtime-dispatched-moutline-atomicshelper calls (__aarch64_cas*,__aarch64_ldadd*,__aarch64_swp*) on baseline-march=armv8-abuilds.dmb ish):Barrier_AtomicIncrement,Acquire_CompareAndSwap, andRelease_CompareAndSwapwrap relaxed LL/SC loops in fullMemoryBarrier()(dmb ish) fences—even executing a fulldmb ishbarrier whenAcquire_CompareAndSwapfails—rather than using architectural C++11/ARMv8 acquire/release semantics (ldaxr/stlxron ARMv8.0-A orcasa/casl/ldaddalon ARMv8.1-A+ LSE).<libkern/OSAtomic.h>Fallback on Apple Silicon:In
src/butil/atomicops.h,#elif defined(OS_MACOSX)precedes#elif defined(COMPILER_GCC) && defined(ARCH_CPU_ARM64), routing macOS ARM64 builds to the deprecatedOSAtomic*routines inbutil/atomicops_internals_mac.h.What is changed and the side effects?
Changed:
src/butil/atomicops_internals_arm64_gcc.h: Replaced all 32-bit (Atomic32) and 64-bit (Atomic64) inline assembly blocks with GCC/Clang__atomic_*built-ins (__atomic_compare_exchange_n,__atomic_exchange_n,__atomic_add_fetch,__atomic_store_n,__atomic_load_n, and__atomic_thread_fence), preserving Chromium'sAcquire_StoreandRelease_Loadsemantics.src/butil/atomicops.h&BUILD.bazel: Ordered#elif defined(COMPILER_GCC) && defined(ARCH_CPU_ARM64)ahead of#elif defined(OS_MACOSX)so Apple Silicon builds useatomicops_internals_arm64_gcc.h, and exported the header inmacos_lib.hdrsfor Bazel builds on macOS.test/atomicops_unittest.cc: Expanded unit test coverage to exerciseBarrier_AtomicIncrement,Acquire_CompareAndSwap,Release_CompareAndSwap(both success and failure paths), explicitAtomic64instantiations, and multi-threaded CAS/increment contention.Side effects:
Performance effects:
objdump -d):-march=armv8-a(GCC 13.3 / Clang 21): 0 rawldxr/stxrinstructions in caller object files; emits runtime-detected-moutline-atomicshelper calls (__aarch64_cas*,__aarch64_ldadd*,__aarch64_swp*).-march=armv8-a+lse: Emits inlinecas,casa,casl,swp,ldadd, andldaddalinstructions with 0dmb ishbarriers inAcquire_CompareAndSwap,Release_CompareAndSwap, andBarrier_AtomicIncrement.ldxr/stxr+dmb ish)-march=armv8-aoutline LSE)-march=armv8-a+lseinline LSE)Barrier_AtomicIncrementAcquire_CAS+Release_Store)Breaking backward compatibility: No. Preserves 100% API and ABI compatibility for
butil::subtle::Atomic32,Atomic64, andAtomicWord.How to Validate Correctness (Disassembly & Unit Tests)
Reproducible
objdump -dverification and unit test commands + captured output1. Verify LSE instruction generation (
ldadd,ldaddal,casa,casl,ldapr/ldar,stlr) with zerodmb ishbarriersRun from the repository root on any ARM64 Linux or macOS machine:
Captured output:
2. Run the unit test suite
Check List: