Repository navigation
perf(butil): add ARM64 hardware CRC32C (+crc and PMULL) acceleration - #3754
Open
jimmystewpot wants to merge 4 commits into
Open
jimmystewpot wants to merge 4 commits into
jimmystewpot wants to merge 4 commits into
Conversation
…hmark tests Signed-off-by: James Lamb <[email protected]>
Signed-off-by: James Lamb <[email protected]>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Critical unresolved issues remain: capability-probe failures may enable unsupported CRC32 instructions, and baseline GCC builds may lack the required intrinsic declarations.
2 open issues
What changed in this PR
Adds runtime-dispatched ARM64 CRC32C acceleration to butil, using CRC32 instructions and optional PMULL folding while preserving the existing software fallback and API behavior.
Changes:
- Adds Linux and macOS ARM64 capability detection with CRC32 and PMULL implementations.
- Integrates ARM64 runtime dispatch into CRC32C extension and fast-support checks.
- Expands tests for reference validation, alignments, tails, incremental updates, large buffers, and throughput benchmarking.
| File | Description |
|---|---|
| test/crc32c_unittest.cc | Adds ARM64 CRC32C correctness tests covering alignments, tail lengths, large buffers, incremental extensions, capability checks, and throughput benchmarking. |
| src/butil/crc32c.cc | Implements ARM64 CRC32 and PMULL acceleration with runtime feature detection, pipelined folding, dispatch selection, and software fallback. |
Review effort: Lite
…OS sysctl probes Signed-off-by: James Lamb <[email protected]>
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Approval should be deferred because the ARM64 ACLE/NEON target-specific code and CRC/PMULL runtime dispatch require native GCC/Clang builds and execution on representative Linux and macOS AArch64 systems that are unavailable here.
No open issues 🎉
2 resolved since last review
Review effort: Lite
…and unsigned shifts Signed-off-by: James Lamb <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

What problem does this PR solve?
Issue Number: #3753
Problem Summary:
This issue was discovered while profiling StarRocks (which relies heavily on
brpc/bthread/butilfor RPC and data framing) on ARM64 (__aarch64__) servers. Although it doesn't use this library's CRC, it was what prompted me to investigate the current state of ARM/NEON in this codebase.src/butil/crc32c.cccurrently accelerates CRC32C on x86_64 (__SSE4_2__) and RISC-V (__riscv_zbc/__riscv_zvbc), but has no ARM64 (__aarch64__) hardware path. On ARM64:butil::crc32c::IsFastCrc32Supported()returnsfalse.butil::crc32c::Extend()falls back toExtendImpl<SlowCRC32Functor>, a 4-byte-at-a-time software lookup table (~1.1–1.2 GiB/s), despite hardwareFEAT_CRC32andFEAT_PMULLbeing standard across modern ARM64 server and desktop CPUs (ARM Neoverse N1/N2/V1/V2, AWS Graviton 2/3/4, Ampere Altra, and Apple Silicon).What is changed and the side effects?
Changed:
src/butil/crc32c.cc:#if defined(__aarch64__)hardware-accelerated CRC32C path using<arm_acle.h>(__crc32cb,__crc32ch,__crc32cw,__crc32cd) and<arm_neon.h>(vmull_p64).__attribute__((target("+crc")))/__attribute__((target("+crc+crypto")))on GCC, and__attribute__((target("crc")))/__attribute__((target("crc,aes")))on Clang) so that default baseline-march=armv8-abuilds compile cleanly with zero build-flag changes required.isArmCrc32()andisArmPmull()) usinggetauxval(AT_HWCAP)(HWCAP_CRC32andHWCAP_PMULL) on Linux andsysctlbynameon macOS, wired intoChoose_Extend()andIsFastCrc32Supported()so a single baseline binary automatically selects the fastest supported path (CRC32+PMULL, scalarCRC32, or software table fallback) at runtime.__crc32cdlanes combined viavmull_p64carry-less multiplication across1008-byte (3 x 336B) and192-byte (3 x 64B) blocks whenFEAT_PMULLis available, with scalar__crc32cd/__crc32cw/__crc32ch/__crc32cbhandling short buffers, unaligned prologues, and tails.test/crc32c_unittest.cc:IsFastCrc32Supported()on AArch64, bit-for-bit agreement against an independent bitwise Castagnoli reference across alignments0..15, all tail lengths0..512, stripe boundaries (191..3073), large buffers (1 KiB..64 KiB), and multi-chunk incrementalExtend()calls, as well as a throughput benchmark.Side effects:
Performance effects:
-march=armv8-a):SlowCRC32Functorsoftware table)+crc+crypto)Breaking backward compatibility: No. Preserves 100% API, ABI, and wire-format compatibility (
CRC.*,brpc_checksum_unittest, andRecordIOTest.*all pass).How to Validate Correctness (Disassembly & Unit Tests)
Reproducible
objdump -dbinary equivalence checks and unit test commands + captured output1. Verify baseline
-march=armv8-avs-march=armv8-a+crc+cryptoproduces 100% identical object codeBecause
Arm64ExtendCrc32andArm64ExtendCrc32Pmulluse per-function target attributes, compilingsrc/butil/crc32c.ccwith baseline-march=armv8-aemits the exact same machine code as compiling with-march=armv8-a+crc+crypto:2. Inspect generated hardware instructions in
Arm64ExtendCrc32(scalar+crcloop) andArm64ExtendCrc32Pmull(3-way pipelined+crc+pmullSIMD loop)Captured inner-loop disassembly from
/tmp/crc32c_base_armv8a.o:3. Verify software fallback (
ExtendImpl<SlowCRC32Functor>) opcodes are unchanged frommasterComparing the 32-bit instruction opcode column of
ExtendImpl<SlowCRC32Functor>againstmasterconfirms that the software lookup-table implementation is 100% unchanged:4. Run the unit test suite
Check List: