Skip to content

feat: export the telemetry of the benchmarks over OTLP - #127

Open
lvaroqui wants to merge 3 commits into
mainfrom
cod-4149-featnode-export-the-benchmark-telemetry-from-the-codspeed
Open

lvaroqui wants to merge 3 commits into
mainfrom
cod-4149-featnode-export-the-benchmark-telemetry-from-the-codspeed

Conversation

@lvaroqui

Copy link
Copy Markdown
Contributor

Adds a telemetry runner mode, in which the core exports the OpenTelemetry spans and logs of the benchmarked code over OTLP/HTTP to the receiver at CODSPEED_OTEL_ENDPOINT. CodSpeed can then show the telemetry of each benchmark, as pytest-codspeed and codspeed-rust already do.

How it works

  • Setup. @codspeed/core sets up the export when it is imported, before the benchmarked modules get their tracers and loggers. It registers global tracer and logger providers, and an AsyncLocalStorage context manager that keeps the active span across awaits. It loads the OpenTelemetry SDK with require only when the endpoint is set, so the other modes never load it.
  • Times. Each span and log record carries its times on the clock of the benchmark markers: codspeed.monotonic_start_ns / codspeed.monotonic_end_ns on spans, codspeed.monotonic_time_ns on logs.
  • Rounds. The mode runs each benchmark once, like simulation, but with the default V8 flags, so the timings stay close to production. The tinybench, benchmark.js and Vitest plugins wrap the benchmarked call with withRoundMarkers, which sends the start and end of the call as a round of the benchmark. The runner uses these rounds to find the benchmark of each span and log.
  • Flush. The plugins flush the export once their benchmarks are done, since Vitest may kill its worker before the process exits. With Vitest, a setup file imports the core in each worker, ahead of the benchmark files.

Things worth a look

  • Ids. Trace and span ids come from crypto: the benchmarks run with a seeded Math.random(), so the SDK's default generator would give every process the same ids.
  • Existing providers. If the benchmarked code registered its own providers first, the core keeps them and warns that their telemetry is not exported.
  • Dependencies. The eight @opentelemetry/* packages are regular dependencies of @codspeed/core. Making them optional peer dependencies would keep them out of the default install, at the cost of an install step for telemetry users. @opentelemetry/api in particular is usually a peer dependency, so that the benchmarked code and the core share one copy.
  • Code location. The Node SDK doesn't set code.file.path / code.line.number, so the spans and logs carry no source location yet, unlike the Rust and Python integrations. Adding it means reading the call stack (util.getCallSites()) on each span and log; that can come in a follow-up.

Example

examples/with-telemetry runs the same workloads (nested spans with logs, a log outside any span, a log through @opentelemetry/api-logs, and no telemetry) with each of the three plugins: pnpm bench-vitest, pnpm bench-tinybench, pnpm bench-benchmark-js.

packages/core/tests/telemetry.integ.test.ts runs a script against a local OTLP receiver, and checks the exported spans and logs and their monotonic attributes. It also checks that the SDK isn't loaded without an endpoint.

Closes COD-4149

@lvaroqui
lvaroqui marked this pull request as ready for review November 16, 2026 14:19
@greptile-apps

greptile-apps Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The PR adds an opt-in telemetry export path across the Node benchmark runners.

Fix the dependency minimum before merging so installations keep working on supported Node.js 22 releases.

Fix All in Claude CodeFindings

  1. P1 Supported Node.js installs fail ▶
  2. P2 Exported values go unchecked ▶
Fix with agent prompt
### Issue 1
packages/core/package.json:44-49
The new OpenTelemetry SDK packages require Node.js `>=22.15.0`, but this repository still supports earlier Node.js 22 releases. If a user's package manager enforces dependency engine requirements, these regular dependencies prevent them from installing any CodSpeed plugin-even when they never enable telemetry. Use compatible dependencies or explicitly raise the supported minimum.

### Issue 2
packages/core/tests/telemetry.integ.test.ts:108-116
The test searches the protobuf bytes for names and attribute keys, but never checks their values or links. It would pass with zero timestamps, unrelated parent and child spans, or a log missing its span link. Decode the requests and assert timestamp order, the parent-child link, and the log's trace and span IDs.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds a telemetry runner mode that exports benchmark spans and logs over OTLP/HTTP.

  • CodSpeed exports benchmark spans and logs to an OTLP receiver.
  • Benchmark plugins mark each measured call and flush its telemetry.
  • A new example shows spans and logs in all three benchmark plugins.
  • The lockfile's package notices use updated wording.

Diagram

sequenceDiagram
  participant Plugin
  participant Core
  participant Benchmark
  participant Receiver as OTLP receiver
  Plugin->>Core: Import and register providers
  Plugin->>Core: Start sample
  Plugin->>Benchmark: Run with round markers
  Benchmark->>Core: Emit spans and logs
  Core->>Core: Add monotonic timestamps
  Plugin->>Core: Stop sample and flush
  Core->>Receiver: Export traces and logs
Loading

Reviews (2) · Last reviewed commit: "test(core): check the links and times of..." · Reviewed by Greptile

Comment thread packages/core/src/telemetry.ts Resolved
Comment on lines +94 to +102
const traceBody = traces[0].body.toString("latin1");
expect(traceBody).toContain("parent");
expect(traceBody).toContain("child");
expect(traceBody).toContain("codspeed.monotonic_start_ns");
expect(traceBody).toContain("codspeed.monotonic_end_ns");

const logBody = logs[0].body.toString("latin1");
expect(logBody).toContain("in child");
expect(logBody).toContain("codspeed.monotonic_time_ns");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Exported values go unchecked

The test searches the protobuf bytes for names and attribute keys, but never checks their values or links. It would pass with zero timestamps, unrelated parent and child spans, or a log missing its span link. Decode the requests and assert timestamp order, the parent-child link, and the log's trace and span IDs.

Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/core/tests/telemetry.integ.test.ts
Line: 94-102

Comment:
**Exported values go unchecked**

The test searches the protobuf bytes for names and attribute keys, but never checks their values or links. It would pass with zero timestamps, unrelated parent and child spans, or a log missing its span link. Decode the requests and assert timestamp order, the parent-child link, and the log's trace and span IDs.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Claude Code Fix in Codex

lvaroqui and others added 3 commits November 16, 2026 15:43
Add a `telemetry` runner mode. In this mode, the runner sets
`CODSPEED_OTEL_ENDPOINT`, and the core sends the OpenTelemetry spans and
logs of the benchmarked code over OTLP/HTTP to that receiver, so that
CodSpeed can show the telemetry of each benchmark.

The core sets up the export when it is imported, before the benchmarked
modules get their tracers and loggers. It registers global tracer and
logger providers and an `AsyncLocalStorage` context manager, which keeps
the active span across async calls. It loads the OpenTelemetry SDK only
when the endpoint is set, so the other modes never load it. If the
benchmarked code already registered its own providers, the core keeps
them and warns that their telemetry is not exported.

Each span and log record also carries its times on the clock of the
benchmark markers (`codspeed.monotonic_start_ns`,
`codspeed.monotonic_end_ns`, `codspeed.monotonic_time_ns`). Trace and
span ids come from `crypto`, because the benchmarks run with a seeded
`Math.random()`.

The mode runs each benchmark once, like simulation, but with the default
V8 flags, so the timings stay close to production. The tinybench,
benchmark.js and Vitest plugins wrap each benchmarked call with
`withRoundMarkers`, which sends the start and end of the call as a round
of the benchmark. The runner uses these rounds to find the benchmark of
each span and log. The plugins also flush the export once their
benchmarks are done, since Vitest may kill its worker before the process
exits. With Vitest, a setup file imports the core in each worker, ahead
of the benchmark files.

Add the `examples/with-telemetry` project, which runs the same workloads
with each of the three plugins.

Closes COD-4149
Co-Authored-By: Claude Opus 5.5 <[email protected]>
…runs

`withRoundMarkers` only waited for native promises. A benchmark returning
another thenable, such as a database query, got its end marker as soon as
it returned, so the spans and logs it emitted afterwards fell outside its
round. It now waits for any object with a `then` method, and returns a
native promise, which the plugins only await.

The Vitest runners skipped `flushTelemetry()` when a benchmark or a hook
threw, so Vitest could stop the worker with spans and logs still queued,
including those that explain the failure. They now flush in a `finally`
block.

Refs COD-4149
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The integration test only searched the exported protobuf bytes for names
and attribute keys: it would pass with zero timestamps, unlinked spans or
a log missing its span.

Split the building of the providers out of `registerProviders` into
`buildProviders`, which takes the span and log exporters, and test it
with the SDK's in-memory exporters. The test checks that the child span
links to its parent, that a log links to its span, and that the monotonic
times of the spans and logs are in order. The subprocess test still
covers the OTLP/HTTP export.

Refs COD-4149
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@lvaroqui
lvaroqui force-pushed the cod-4149-featnode-export-the-benchmark-telemetry-from-the-codspeed branch from a81d3d5 to 5fc9260 Compare November 16, 2026 14:48
Comment on lines +44 to +49
"@opentelemetry/context-async-hooks": "^3.1.0",
"@opentelemetry/exporter-logs-otlp-proto": "^0.301.0",
"@opentelemetry/exporter-trace-otlp-proto": "^0.301.0",
"@opentelemetry/resources": "^3.1.0",
"@opentelemetry/sdk-logs": "^3.1.0",
"@opentelemetry/sdk-trace": "^3.1.0",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Supported Node.js installs fail

The new OpenTelemetry SDK packages require Node.js >=22.15.0, but this repository still supports earlier Node.js 22 releases. If a user's package manager enforces dependency engine requirements, these regular dependencies prevent them from installing any CodSpeed plugin—even when they never enable telemetry. Use compatible dependencies or explicitly raise the supported minimum.

Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/core/package.json
Line: 44-49

Comment:
**Supported Node.js installs fail**

The new OpenTelemetry SDK packages require Node.js `>=22.15.0`, but this repository still supports earlier Node.js 22 releases. If a user's package manager enforces dependency engine requirements, these regular dependencies prevent them from installing any CodSpeed plugin—even when they never enable telemetry. Use compatible dependencies or explicitly raise the supported minimum.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Claude Code Fix in Codex

@codspeed

codspeed Bot commented Nov 16, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 11.51%

4 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 14 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ 17 benchmarks with a significant change ran on different hardware

Their base and head results were measured on machines with a different CPU, core count or memory, so the changes might not come from the code. Run base and head on the same runner type.

Learn more about Macro Runners

See affected benchmarks

⚠️ 16 benchmarks with a significant change ran in a different software environment

Their base and head results were measured with a different OS or toolchain, such as compiler, runtime or library versions, so the changes might come from this difference rather than from the code. Open an affected benchmark to see what changed.

See affected benchmarks

ℹ️ 18 benchmarks have no flamegraph

The flamegraph wasn't generated: most of the time was spent in runtime internals, which would leave nothing useful to analyze.

See affected benchmarks

⚡ 7 improved benchmarks
❌ 10 regressed benchmarks
✅ 253 untouched benchmarks
🆕 32 new benchmarks
⏩ 56 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ CPU Simulation recursive fibo 10 334.6 µs 679.1 µs -50.73%
❌ Memory test_recursive_cached_fibo_20 784 B 1,312 B -40.24%
❌ Memory test_recursive_cached_fibo_30 784 B 1,312 B -40.24%
❌ Memory test_recursive_fibo_20 784 B 1,312 B -40.24%
❌ Memory test_recursive_cached_fibo_10 784 B 1,312 B -40.24%
❌ Memory test sync baz 100 784 B 1,312 B -40.24%
❌ Walltime fibo darwin 5.2 ms 8.2 ms -35.97%
❌ CPU Simulation recursive fibo 10 333.8 µs 508.6 µs -34.37%
❌ Memory wait 500ms 22.7 KB 28.7 KB -20.97%
❌ CPU Simulation switch 1 526.8 µs 650.6 µs -19.02%
⚡ CPU Simulation wait 1sec 98.8 ms 58.9 ms +67.81%
⚡ CPU Simulation wait 500ms 46 ms 29.6 ms +55.49%
⚡ CPU Simulation test_recursive_fibo_10 510 µs 336.4 µs +51.62%
⚡ Memory count is back to zero 2.1 KB 1.5 KB +44.09%
⚡ Memory one 2 KB 1.5 KB +35.48%
⚡ CPU Simulation wait 500ms 38.9 ms 28.9 ms +34.67%
⚡ CPU Simulation wait 1sec 72.9 ms 61.2 ms +19.18%
🆕 Walltime legacy_log N/A 84 ns N/A
🆕 Walltime log_without_span N/A 84 ns N/A
🆕 Walltime no_telemetry N/A 84 ns N/A
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing cod-4149-featnode-export-the-benchmark-telemetry-from-the-codspeed (5fc9260) with main (bfc3238)

Open in CodSpeed

Footnotes

  1. 56 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant