Skip to content

perf(butil): serialize ARM64 clock_cycles with isb and use fixed-point scaling in cpuwide_time_ns - #3756

Open
jimmystewpot wants to merge 8 commits into
apache:masterfrom
jimmystewpot:perf/arm64-fast-timer
Open

jimmystewpot wants to merge 8 commits into
apache:masterfrom
jimmystewpot:perf/arm64-fast-timer

Conversation

@jimmystewpot

@jimmystewpot jimmystewpot commented Nov 18, 2026 •

Copy link
Copy Markdown

What problem does this PR solve?

Issue Number: #3753

Problem Summary:

This issue was discovered while auditing and profiling brpc / bthread / butil timer overhead for StarRocks on ARM64 (__aarch64__) servers. Even though StarRocks' default build configuration uses WITH_CPU_FREQUENCY=OFF, evaluating WITH_CPU_FREQUENCY=ON on ARM64 (where CNTFRQ_EL0 provides an architectural, invariant counter frequency) revealed two defects in src/butil/time.h and src/butil/time.cpp:

  1. Unserialised cntvct_el0 reads in butil::detail::clock_cycles():
    On __aarch64__, clock_cycles() executes mrs %0, cntvct_el0 without an isb barrier. Per the ARM Architecture Reference Manual (DDI 0487, Chapter D12 "The Generic Timer in AArch64 state", Section D12.2.2 "The virtual counter"), reads of CNTVCT_EL0 can execute speculatively and out-of-order relative to earlier loads, stores, or conditional branches unless preceded by an isb. Without serialisation, short timed intervals (butil::Timer) can read cntvct_el0 before an earlier memory access or lock acquisition retires.
  2. Two dependent 64-bit integer divisions on every cpuwide_time_ns() call:
    cpuwide_time_ns() converts clock_cycles() ticks to nanoseconds by executing two runtime 64-bit integer divisions by detail::invariant_cpu_freq (tsc / cpu_freq and remain * 1000000000L / cpu_freq) on every call, rather than precomputing a fixed-point multiplier and shift alongside invariant_cpu_freq at initialisation time.

What is changed and the side effects?

Changed:

  • src/butil/time.h & src/butil/time.cpp:
    • Preceded mrs %0, cntvct_el0 with isb (and a "memory" clobber) in butil::detail::clock_cycles() on __aarch64__ to prevent out-of-order speculative counter hoisting.
    • Added butil::detail::compute_cpu_frequency_scale() and butil::detail::cycles_to_ns() (guarded by #if defined(__SIZEOF_INT128__) with a division-based fallback when __SIZEOF_INT128__ is unavailable), precomputing invariant_cpu_freq_mult and invariant_cpu_freq_shift inside read_invariant_cpu_frequency() before publishing invariant_cpu_freq.
    • Updated cpuwide_time_ns() to convert cycle counts via 128-bit mul + shift (cycles_to_ns), eliminating both runtime 64-bit integer divisions on 64-bit platforms while preserving exact nanosecond monotonicity and overflow safety across decades of uptime.
  • test/baidu_time_unittest.cpp:
    • Added BaiduTimeTest.clock_cycles_monotonic_and_serialized (verifying back-to-back monotonicity and cross-thread acquire/release ordering across 2,000 lock-step handoffs on AArch64).
    • Added BaiduTimeTest.cpu_frequency_scale_accuracy_and_monotonicity (verifying <= 1 ns accuracy against exact division and strict monotonicity across standard ARM64 and x86_64 timer frequencies from 19.2 MHz to 3.59 GHz and uptimes up to $10^8$ seconds).
    • Added BaiduTimeTest.cpuwide_time_with_cpu_frequency (verifying cycle-bracketed cpuwide_time_ns() equivalence, CNTFRQ_EL0 wall-clock agreement, and scaling latency).

Side effects:

  • Performance effects:

    • Scaling Arithmetic Latency (ARM Neoverse-V2, Ubuntu 24.04, GCC 13.3.0):
      • Reduces cycle-to-nanosecond scaling math latency from 7.91 ns / op (two 64-bit udiv instructions) to 3.04 ns / op (mul/umulh + lsr — 2.6x faster), and reduces end-to-end serialised cpuwide_time_ns() latency to 23.71 ns / call vs 33.36 ns / call for Linux vDSO monotonic_time_ns() (29% faster).
    • macOS Apple Silicon (BaiduTimeTest.cost_of_timer & cpuwide_time_with_cpu_frequency, Apple Clang 21):
      • Reduces butil::cpuwide_time_ns() and butil::Timer::stop() from 18 ns / call (clock_gettime) to 8–9 ns / call (2x faster), and reduces pure scaling math from 6 ns / op (2x udiv) to 2 ns / op (mul + shift — 3x faster).
  • Breaking backward compatibility: No. butil::detail::invariant_cpu_freq (int64_t) and all public butil timer APIs are preserved with 100% API and ABI compatibility.

How to Validate Correctness (Disassembly & Unit Tests)

Reproducible objdump -d verification and unit test commands + captured output

1. Verify clock_cycles() emits isb; mrs x0, CNTVCT_EL0 and cycles_to_ns() emits mul/umulh + shifts (zero udiv instructions)

Run from the repository root on any ARM64 Linux or macOS machine:

c++ -O2 -std=c++17 -DBUTIL_USE_CPU_FREQUENCY=1 -Isrc -x c++ - -c -o /tmp/fast_timer_check.o << 'CPP' && \
objdump -d --no-leading-addr /tmp/fast_timer_check.o && rm -f /tmp/fast_timer_check.o
#include "butil/time.h"
extern "C" {
uint64_t check_clock_cycles() { return butil::detail::clock_cycles(); }
int64_t check_cycles_to_ns(uint64_t c, uint64_t m, uint32_t s) { return butil::detail::cycles_to_ns(c, m, s); }
}
CPP

Captured output:

<_check_clock_cycles>:
 d5033fdf   isb
 d53be040   mrs    x0, CNTVCT_EL0
 d65f03c0   ret

<_check_cycles_to_ns>:
 9b007c28   mul    x8, x1, x0
 9bc07c29   umulh  x9, x1, x0
 2a2203ea   mvn    w10, w2
 d37ff92b   lsl    x11, x9, #1
 9aca216a   lsl    x10, x11, x10
 2a0203eb   mov    w11, w2
 9acb2508   lsr    x8, x8, x11
 aa080148   orr    x8, x10, x8
 9acb2529   lsr    x9, x9, x11
 f27a017f   tst    x11, #0x40
 9a881120   csel   x0, x9, x8, ne
 d65f03c0   ret

2. Run the unit test suite (with both WITH_CPU_FREQUENCY=OFF and WITH_CPU_FREQUENCY=ON)

cmake -B build -DBUILD_UNIT_TESTS=ON -DWITH_CPU_FREQUENCY=ON && cmake --build build --target test_butil -j6
./build/test/test_butil --gtest_filter='BaiduTimeTest.*'

Check List:

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The change still has an incorrect no-int128 conversion fallback and a test failure path that can call std::terminate before joining the producer.

2 open issues
What changed in this PR

This PR improves ARM64 butil timer correctness and performance by serializing virtual counter reads and replacing repeated frequency divisions with precomputed fixed-point scaling.

Changes:

  • Adds an ARM64 isb barrier and memory clobber before cntvct_el0 reads.
  • Precomputes frequency scaling factors for cycle-to-nanosecond conversion.
  • Adds tests for counter ordering, scaling accuracy, monotonicity, and CPU-frequency timing.
File Description
test/​baidu_time_unittest.cpp Adds tests covering counter ordering, scaling accuracy, monotonicity, and CPU-frequency-based timing behavior.
src/​butil/​time.h Serializes ARM64 counter reads and introduces fixed-point cycle-to-nanosecond conversion helpers.
src/​butil/​time.cpp Computes and initializes the frequency scaling factors alongside the invariant CPU frequency.

Review effort: Lite

Comment thread src/butil/time.h Resolved
Comment thread test/baidu_time_unittest.cpp Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Approval is deferred because the AArch64 inline assembly and timing calibration require runtime validation on representative ARM64 Linux and Apple Silicon systems that is unavailable in this review environment.

No open issues 🎉

2 resolved since last review

Review effort: Lite

…ests and unused var in time.cpp

Signed-off-by: James Lamb <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(butil): ARM64 (aarch64) performance and modernisation improvements across atomics, CRC32C, spin-wait hints, and fast timers

2 participants