Repository navigation
perf(butil): serialize ARM64 clock_cycles with isb and use fixed-point scaling in cpuwide_time_ns - #3756
Open
jimmystewpot wants to merge 8 commits into
Open
perf(butil): serialize ARM64 clock_cycles with isb and use fixed-point scaling in cpuwide_time_ns#3756jimmystewpot wants to merge 8 commits into
jimmystewpot wants to merge 8 commits into
Conversation
…cpuwide_time_ns Signed-off-by: James Lamb <[email protected]>
…t scaling in cpuwide_time_ns Signed-off-by: James Lamb <[email protected]>
Signed-off-by: James Lamb <[email protected]>
…tforms Signed-off-by: James Lamb <[email protected]>
…checks Signed-off-by: James Lamb <[email protected]>
…test Signed-off-by: James Lamb <[email protected]>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The change still has an incorrect no-int128 conversion fallback and a test failure path that can call std::terminate before joining the producer.
2 open issues
What changed in this PR
This PR improves ARM64 butil timer correctness and performance by serializing virtual counter reads and replacing repeated frequency divisions with precomputed fixed-point scaling.
Changes:
- Adds an ARM64
isbbarrier and memory clobber beforecntvct_el0reads. - Precomputes frequency scaling factors for cycle-to-nanosecond conversion.
- Adds tests for counter ordering, scaling accuracy, monotonicity, and CPU-frequency timing.
| File | Description |
|---|---|
| test/baidu_time_unittest.cpp | Adds tests covering counter ordering, scaling accuracy, monotonicity, and CPU-frequency-based timing behavior. |
| src/butil/time.h | Serializes ARM64 counter reads and introduces fixed-point cycle-to-nanosecond conversion helpers. |
| src/butil/time.cpp | Computes and initializes the frequency scaling factors alongside the invariant CPU frequency. |
Review effort: Lite
… join producer on failure Signed-off-by: James Lamb <[email protected]>
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Approval is deferred because the AArch64 inline assembly and timing calibration require runtime validation on representative ARM64 Linux and Apple Silicon systems that is unavailable in this review environment.
No open issues 🎉
2 resolved since last review
Review effort: Lite
…ests and unused var in time.cpp Signed-off-by: James Lamb <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

What problem does this PR solve?
Issue Number: #3753
Problem Summary:
This issue was discovered while auditing and profiling
brpc/bthread/butiltimer overhead for StarRocks on ARM64 (__aarch64__) servers. Even though StarRocks' default build configuration usesWITH_CPU_FREQUENCY=OFF, evaluatingWITH_CPU_FREQUENCY=ONon ARM64 (whereCNTFRQ_EL0provides an architectural, invariant counter frequency) revealed two defects insrc/butil/time.handsrc/butil/time.cpp:cntvct_el0reads inbutil::detail::clock_cycles():On
__aarch64__,clock_cycles()executesmrs %0, cntvct_el0without anisbbarrier. Per the ARM Architecture Reference Manual (DDI 0487, Chapter D12 "The Generic Timer in AArch64 state", Section D12.2.2 "The virtual counter"), reads ofCNTVCT_EL0can execute speculatively and out-of-order relative to earlier loads, stores, or conditional branches unless preceded by anisb. Without serialisation, short timed intervals (butil::Timer) can readcntvct_el0before an earlier memory access or lock acquisition retires.cpuwide_time_ns()call:cpuwide_time_ns()convertsclock_cycles()ticks to nanoseconds by executing two runtime 64-bit integer divisions bydetail::invariant_cpu_freq(tsc / cpu_freqandremain * 1000000000L / cpu_freq) on every call, rather than precomputing a fixed-point multiplier and shift alongsideinvariant_cpu_freqat initialisation time.What is changed and the side effects?
Changed:
src/butil/time.h&src/butil/time.cpp:mrs %0, cntvct_el0withisb(and a"memory"clobber) inbutil::detail::clock_cycles()on__aarch64__to prevent out-of-order speculative counter hoisting.butil::detail::compute_cpu_frequency_scale()andbutil::detail::cycles_to_ns()(guarded by#if defined(__SIZEOF_INT128__)with a division-based fallback when__SIZEOF_INT128__is unavailable), precomputinginvariant_cpu_freq_multandinvariant_cpu_freq_shiftinsideread_invariant_cpu_frequency()before publishinginvariant_cpu_freq.cpuwide_time_ns()to convert cycle counts via 128-bitmul + shift(cycles_to_ns), eliminating both runtime 64-bit integer divisions on 64-bit platforms while preserving exact nanosecond monotonicity and overflow safety across decades of uptime.test/baidu_time_unittest.cpp:BaiduTimeTest.clock_cycles_monotonic_and_serialized(verifying back-to-back monotonicity and cross-thread acquire/release ordering across 2,000 lock-step handoffs on AArch64).BaiduTimeTest.cpu_frequency_scale_accuracy_and_monotonicity(verifying<= 1ns accuracy against exact division and strict monotonicity across standard ARM64 and x86_64 timer frequencies from 19.2 MHz to 3.59 GHz and uptimes up toBaiduTimeTest.cpuwide_time_with_cpu_frequency(verifying cycle-bracketedcpuwide_time_ns()equivalence,CNTFRQ_EL0wall-clock agreement, and scaling latency).Side effects:
Performance effects:
udivinstructions) to 3.04 ns / op (mul/umulh+lsr— 2.6x faster), and reduces end-to-end serialisedcpuwide_time_ns()latency to 23.71 ns / call vs 33.36 ns / call for Linux vDSOmonotonic_time_ns()(29% faster).BaiduTimeTest.cost_of_timer&cpuwide_time_with_cpu_frequency, Apple Clang 21):butil::cpuwide_time_ns()andbutil::Timer::stop()from 18 ns / call (clock_gettime) to 8–9 ns / call (2x faster), and reduces pure scaling math from 6 ns / op (2x udiv) to 2 ns / op (mul + shift— 3x faster).Breaking backward compatibility: No.
butil::detail::invariant_cpu_freq(int64_t) and all publicbutiltimer APIs are preserved with 100% API and ABI compatibility.How to Validate Correctness (Disassembly & Unit Tests)
Reproducible
objdump -dverification and unit test commands + captured output1. Verify
clock_cycles()emitsisb; mrs x0, CNTVCT_EL0andcycles_to_ns()emitsmul/umulh+ shifts (zeroudivinstructions)Run from the repository root on any ARM64 Linux or macOS machine:
Captured output:
2. Run the unit test suite (with both
WITH_CPU_FREQUENCY=OFFandWITH_CPU_FREQUENCY=ON)Check List: