Skip to content

feat(criterion): export the telemetry of benchmarks over OTLP - #204

Draft
lvaroqui wants to merge 1 commit into
mainfrom
cod-4287-telemetry-export-in-codspeed-criterion-compat
Draft

lvaroqui wants to merge 1 commit into
mainfrom
cod-4287-telemetry-export-in-codspeed-criterion-compat

Conversation

@lvaroqui

Copy link
Copy Markdown
Contributor

Export the traces and logs that Criterion.rs benchmarks emit with tracing and log, as #197 did for divan. Using it takes the telemetry feature of codspeed-criterion-compat and a telemetry build:

cargo codspeed build -m telemetry            # or with other modes, e.g. -m simulation,telemetry
codspeed run -m telemetry -- cargo codspeed run

When CODSPEED_OTEL_ENDPOINT is set, the main of criterion_main! installs the tracing subscriber of codspeed, which exports spans as traces, and tracing events and log records as logs, and keeps it until the groups are done. The export, its filter (CODSPEED_TELEMETRY_FILTER) and its timing attributes are the ones of codspeed, unchanged: only the telemetry build of cargo-codspeed compiles them.

Worth a careful look

  • criterion_main! expands in the user's crate, where #[cfg(feature = "telemetry")] would read the user's features, not ours. So the macro calls a hidden __init_telemetry() of the compat crate, which returns the guard with the feature and None without it.
  • The macro of the Criterion fork, used outside of cargo codspeed, is unchanged.
  • A custom main (the Criterion::new_instrumented() pattern) calls criterion::telemetry::init() itself and keeps the guard, as a divan benchmark with its own subscriber adds divan::telemetry::layer(). This is for the docs, along with lifting their divan-only note.

crates/criterion_compat/benches/telemetry.rs emits telemetry in each of the ways the divan example does. It needs the feature (required-features), so cargo codspeed build -p codspeed-criterion-compat without --features telemetry, as in CI, skips it.

Checked end to end with a telemetry build of that bench run against a local OTLP/HTTP receiver: the info spans (parse_and_sum, parse, sum) arrive on /v1/traces, the events and log records on /v1/logs, and the debug ones are filtered out. Without CODSPEED_OTEL_ENDPOINT, nothing is sent.

Closes COD-4287

Add a `telemetry` feature to `codspeed-criterion-compat`, as
`codspeed-divan-compat` has. When `CODSPEED_OTEL_ENDPOINT` is set, the
`main` of `criterion_main!` installs the `tracing` subscriber of
`codspeed`, which exports spans as traces, and `tracing` events and
`log` records as logs, and keeps it until the groups are done. The
export is the one of the `codspeed` crate, so only the telemetry build
of `cargo-codspeed` compiles it.

The macro expands in the user's crate, where it can't check the feature
of the compat crate, so it calls a hidden function that is a no-op
without the feature. `criterion::telemetry` re-exports the module, for
custom `main` functions and benchmarks installing their own subscriber.

`benches/telemetry.rs` emits telemetry in each way the divan example
does. It needs the feature, so builds without it skip it.

Closes COD-4287
Co-Authored-By: Claude Fable 5.1 <[email protected]>
@codspeed

codspeed Bot commented Nov 16, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 4.78%

4 warnings, 1 note

⚠️ Unable to measure the execution of 59 benchmarks

These benchmarks measured no execution time, so their results are not comparable and count as unchanged. This usually happens when the compiler removes the code under test.

Prevent compiler optimizations

See affected benchmarks

⚠️ 17 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ 1 benchmark with a significant change ran on different hardware

Its base and head results were measured on machines with a different CPU, core count or memory, so the change might not come from the code. Run base and head on the same runner type.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark with a significant change ran in a different software environment

Its base and head results were measured with a different OS or toolchain, such as compiler, runtime or library versions, so the change might come from this difference rather than from the code. Open the benchmark to see what changed.

See affected benchmarks

ℹ️ 57 benchmarks have no flamegraph

  • The flamegraph wasn't generated: most of the time was spent in runtime internals, which would leave nothing useful to analyze. (9 benchmarks)
  • The flamegraph wasn't generated: no execution data was recorded while the benchmark ran. Make sure the compiler didn't optimize the benchmark out. (48 benchmarks)

See affected benchmarks

⚡ 8 improved benchmarks
❌ 8 regressed benchmarks
✅ 602 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Walltime parse_and_sum 275 ns 791 ns -65.23%
❌ Walltime hamiltonian_cycle[6] 908 ns 1,169 ns -22.33%
❌ Walltime iter_batched_per_iteration 48 ns 59 ns -18.64%
❌ Walltime rem 5 ns 6 ns -16.67%
❌ Walltime from_elem_decimal[1024] 201 ns 230 ns -12.61%
❌ Walltime graph_coloring[6] 1.2 µs 1.3 µs -11.12%
❌ Walltime recursive[10] 421 ns 443 ns -4.97%
❌ Walltime recursive_memoized[HashMap<u64, u64>, 30] 6.6 µs 6.8 µs -3.52%
⚡ Walltime hamiltonian_cycle[4] 1,295 ns 785 ns +64.97%
⚡ Walltime bench_array1[10] 6 ns 5 ns +20%
⚡ Walltime iter_batched_large_input 8 ns 7 ns +14.29%
⚡ Walltime spawn_on_parent_runtime 24.5 µs 21.9 µs +11.54%
⚡ Walltime graph_coloring[5] 1.2 µs 1.1 µs +11.18%
⚡ Walltime recursive_memoized[HashMap<u64, u64>, 10] 2.1 µs 1.9 µs +9.62%
⚡ CPU Simulation spawn_on_parent_runtime 36.2 µs 33.3 µs +8.47%
⚡ Walltime iter_with_setup 42 ns 40 ns +5%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing cod-4287-telemetry-export-in-codspeed-criterion-compat (4dba4f7) with main (aa1bc87)

Open in CodSpeed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant