Skip to content

perf(butil): add ARM64 hardware CRC32C (+crc and PMULL) acceleration - #3754

Open
jimmystewpot wants to merge 4 commits into
apache:masterfrom
jimmystewpot:perf/arm64-crc32c
Open

jimmystewpot wants to merge 4 commits into
apache:masterfrom
jimmystewpot:perf/arm64-crc32c

Conversation

@jimmystewpot

@jimmystewpot jimmystewpot commented Nov 18, 2026 •

Copy link
Copy Markdown

What problem does this PR solve?

Issue Number: #3753

Problem Summary:

This issue was discovered while profiling StarRocks (which relies heavily on brpc / bthread / butil for RPC and data framing) on ARM64 (__aarch64__) servers. Although it doesn't use this library's CRC, it was what prompted me to investigate the current state of ARM/NEON in this codebase.

src/butil/crc32c.cc currently accelerates CRC32C on x86_64 (__SSE4_2__) and RISC-V (__riscv_zbc / __riscv_zvbc), but has no ARM64 (__aarch64__) hardware path. On ARM64:

  • butil::crc32c::IsFastCrc32Supported() returns false.
  • butil::crc32c::Extend() falls back to ExtendImpl<SlowCRC32Functor>, a 4-byte-at-a-time software lookup table (~1.1–1.2 GiB/s), despite hardware FEAT_CRC32 and FEAT_PMULL being standard across modern ARM64 server and desktop CPUs (ARM Neoverse N1/N2/V1/V2, AWS Graviton 2/3/4, Ampere Altra, and Apple Silicon).

What is changed and the side effects?

Changed:

  • src/butil/crc32c.cc:
    • Added an #if defined(__aarch64__) hardware-accelerated CRC32C path using <arm_acle.h> (__crc32cb, __crc32ch, __crc32cw, __crc32cd) and <arm_neon.h> (vmull_p64).
    • Annotated the hardware routines with per-function compiler target attributes (__attribute__((target("+crc"))) / __attribute__((target("+crc+crypto"))) on GCC, and __attribute__((target("crc"))) / __attribute__((target("crc,aes"))) on Clang) so that default baseline -march=armv8-a builds compile cleanly with zero build-flag changes required.
    • Added cached runtime CPU capability detection (isArmCrc32() and isArmPmull()) using getauxval(AT_HWCAP) (HWCAP_CRC32 and HWCAP_PMULL) on Linux and sysctlbyname on macOS, wired into Choose_Extend() and IsFastCrc32Supported() so a single baseline binary automatically selects the fastest supported path (CRC32+PMULL, scalar CRC32, or software table fallback) at runtime.
    • Implemented 3-way pipelined __crc32cd lanes combined via vmull_p64 carry-less multiplication across 1008-byte (3 x 336B) and 192-byte (3 x 64B) blocks when FEAT_PMULL is available, with scalar __crc32cd/__crc32cw/__crc32ch/__crc32cb handling short buffers, unaligned prologues, and tails.
  • test/crc32c_unittest.cc:
    • Added unit tests verifying IsFastCrc32Supported() on AArch64, bit-for-bit agreement against an independent bitwise Castagnoli reference across alignments 0..15, all tail lengths 0..512, stripe boundaries (191..3073), large buffers (1 KiB..64 KiB), and multi-chunk incremental Extend() calls, as well as a throughput benchmark.

Side effects:

  • Performance effects:

    • Linux AArch64 Benchmark (ARM Neoverse-V2, Ubuntu 24.04, GCC 13.3.0, compiled with baseline -march=armv8-a):
      Buffer Size Before (SlowCRC32Functor software table) After (Runtime-detected +crc+crypto) Speedup
      64 B 1,189.4 MiB/s (1.16 GiB/s) 13,015.0 MiB/s (12.71 GiB/s) 10.9x
      256 B 1,215.0 MiB/s (1.19 GiB/s) 16,368.7 MiB/s (15.99 GiB/s) 13.5x
      1 KiB (1024 B) 1,224.1 MiB/s (1.20 GiB/s) 20,323.6 MiB/s (19.85 GiB/s) 16.6x
      4 KiB (4096 B) 1,226.1 MiB/s (1.20 GiB/s) 20,983.9 MiB/s (20.49 GiB/s) 17.1x
      64 KiB (65536 B) 1,225.6 MiB/s (1.20 GiB/s) 21,232.5 MiB/s (20.73 GiB/s) 17.3x
    • macOS Apple Silicon Benchmark (Apple Clang 21):
      • Increases 64 KiB CRC32C throughput from 1,147 MiB/s (1.12 GiB/s) to 28,881 MiB/s (28.20 GiB/s — 25.2x faster).
  • Breaking backward compatibility: No. Preserves 100% API, ABI, and wire-format compatibility (CRC.*, brpc_checksum_unittest, and RecordIOTest.* all pass).

How to Validate Correctness (Disassembly & Unit Tests)

Reproducible objdump -d binary equivalence checks and unit test commands + captured output

1. Verify baseline -march=armv8-a vs -march=armv8-a+crc+crypto produces 100% identical object code

Because Arm64ExtendCrc32 and Arm64ExtendCrc32Pmull use per-function target attributes, compiling src/butil/crc32c.cc with baseline -march=armv8-a emits the exact same machine code as compiling with -march=armv8-a+crc+crypto:

c++ -O2 -std=c++17 -march=armv8-a -Isrc -c src/butil/crc32c.cc -o /tmp/crc32c_base_armv8a.o && \
c++ -O2 -std=c++17 -march=armv8-a+crc+crypto -Isrc -c src/butil/crc32c.cc -o /tmp/crc32c_full_features.o && \
diff -u <(objdump -d --no-leading-addr /tmp/crc32c_base_armv8a.o | sed '1,2d') \
        <(objdump -d --no-leading-addr /tmp/crc32c_full_features.o | sed '1,2d') && \
echo "IDENTICAL: 0 diff lines between -march=armv8-a and -march=armv8-a+crc+crypto"

2. Inspect generated hardware instructions in Arm64ExtendCrc32 (scalar +crc loop) and Arm64ExtendCrc32Pmull (3-way pipelined +crc+pmull SIMD loop)

objdump -d --no-leading-addr /tmp/crc32c_base_armv8a.o

Captured inner-loop disassembly from /tmp/crc32c_base_armv8a.o:

; Scalar +crc 32-byte unrolled loop (Arm64ExtendCrc32):
 a9402829   ldp      x9, x10, [x1]
 9ac95d08   crc32cx  w8, w8, x9
 9aca5d08   crc32cx  w8, w8, x10
 a9412829   ldp      x9, x10, [x1, #0x10]
 9ac95d08   crc32cx  w8, w8, x9
 9aca5d08   crc32cx  w8, w8, x10
 91008021   add      x1, x1, #0x20
 d1008042   sub      x2, x2, #0x20
 f1007c5f   cmp      x2, #0x1f
 54fffee8   b.hi     <Arm64ExtendCrc32+0x34>

; 3-way pipelined +crc + PMULL 1008-byte folding loop (Arm64ExtendCrc32Pmull):
 8b0b002c   add      x12, x1, x11
 f940018d   ldr      x13, [x12]
 9acd5d08   crc32cx  w8, w8, x13          ; lane 0
 f940a98d   ldr      x13, [x12, #0x150]
 9acd5d4a   crc32cx  w10, w10, x13        ; lane 1 (+336B)
 f941518c   ldr      x12, [x12, #0x2a0]
 9acc5d29   crc32cx  w9, w9, x12          ; lane 2 (+672B)
 9100216b   add      x11, x11, #0x8
 f105417f   cmp      x11, #0x150
 54fffee1   b.ne     <Arm64ExtendCrc32Pmull+0x58>
 9e670102   fmov     d2, x8
 0ee0e042   pmull.1q v2, v2, v0           ; fold lane 0 via 64x64->128 carry-less multiply
 9e670143   fmov     d3, x10
 0ee1e063   pmull.1q v3, v3, v1           ; fold lane 1 via 64x64->128 carry-less multiply
 6e221c62   eor.16b  v2, v3, v2
 9e660048   fmov     x8, d2
 9ac85fe8   crc32cx  w8, wzr, x8
 4a090108   eor      w8, w8, w9           ; combine with lane 2

3. Verify software fallback (ExtendImpl<SlowCRC32Functor>) opcodes are unchanged from master

Comparing the 32-bit instruction opcode column of ExtendImpl<SlowCRC32Functor> against master confirms that the software lookup-table implementation is 100% unchanged:

git show master:src/butil/crc32c.cc > /tmp/crc32c_master.cc && \
c++ -O2 -std=c++17 -march=armv8-a -Isrc -c /tmp/crc32c_master.cc -o /tmp/crc32c_master.o && \
diff -u \
  <(objdump -d --no-leading-addr --disassemble-symbols=__ZN5butil6crc32c10ExtendImplINS0_16SlowCRC32FunctorEEEjjPKcm /tmp/crc32c_master.o | awk '/^ [0-9a-f]{8}/{print $1}') \
  <(objdump -d --no-leading-addr --disassemble-symbols=__ZN5butil6crc32c10ExtendImplINS0_16SlowCRC32FunctorEEEjjPKcm /tmp/crc32c_base_armv8a.o | awk '/^ [0-9a-f]{8}/{print $1}') && \
echo "IDENTICAL: Software fallback opcodes are 100% unchanged from master"
rm -f /tmp/crc32c_*.o /tmp/crc32c_master.cc

4. Run the unit test suite

cmake -B build -DBUILD_UNIT_TESTS=ON && cmake --build build --target test_butil -j6
./build/test/test_butil --gtest_filter='CRC*'

Check List:

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical unresolved issues remain: capability-probe failures may enable unsupported CRC32 instructions, and baseline GCC builds may lack the required intrinsic declarations.

2 open issues
What changed in this PR

Adds runtime-dispatched ARM64 CRC32C acceleration to butil, using CRC32 instructions and optional PMULL folding while preserving the existing software fallback and API behavior.

Changes:

  • Adds Linux and macOS ARM64 capability detection with CRC32 and PMULL implementations.
  • Integrates ARM64 runtime dispatch into CRC32C extension and fast-support checks.
  • Expands tests for reference validation, alignments, tails, incremental updates, large buffers, and throughput benchmarking.
File Description
test/​crc32c_unittest.cc Adds ARM64 CRC32C correctness tests covering alignments, tail lengths, large buffers, incremental extensions, capability checks, and throughput benchmarking.
src/​butil/​crc32c.cc Implements ARM64 CRC32 and PMULL acceleration with runtime feature detection, pipelined folding, dispatch selection, and software fallback.

Review effort: Lite

Comment thread src/butil/crc32c.cc Resolved
Comment thread src/butil/crc32c.cc Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Approval should be deferred because the ARM64 ACLE/NEON target-specific code and CRC/PMULL runtime dispatch require native GCC/Clang builds and execution on representative Linux and macOS AArch64 systems that are unavailable here.

No open issues 🎉

2 resolved since last review

Review effort: Lite

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants