Repository navigation
Conversation
[Medium impact] The PR adds an opt-in telemetry export path across the Node benchmark runners. Fix the dependency minimum before merging so installations keep working on supported Node.js 22 releases.
|
| const traceBody = traces[0].body.toString("latin1"); | ||
| expect(traceBody).toContain("parent"); | ||
| expect(traceBody).toContain("child"); | ||
| expect(traceBody).toContain("codspeed.monotonic_start_ns"); | ||
| expect(traceBody).toContain("codspeed.monotonic_end_ns"); | ||
|
|
||
| const logBody = logs[0].body.toString("latin1"); | ||
| expect(logBody).toContain("in child"); | ||
| expect(logBody).toContain("codspeed.monotonic_time_ns"); |
There was a problem hiding this comment.
The test searches the protobuf bytes for names and attribute keys, but never checks their values or links. It would pass with zero timestamps, unrelated parent and child spans, or a log missing its span link. Decode the requests and assert timestamp order, the parent-child link, and the log's trace and span IDs.
Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/core/tests/telemetry.integ.test.ts
Line: 94-102
Comment:
**Exported values go unchecked**
The test searches the protobuf bytes for names and attribute keys, but never checks their values or links. It would pass with zero timestamps, unrelated parent and child spans, or a log missing its span link. Decode the requests and assert timestamp order, the parent-child link, and the log's trace and span IDs.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Add a `telemetry` runner mode. In this mode, the runner sets `CODSPEED_OTEL_ENDPOINT`, and the core sends the OpenTelemetry spans and logs of the benchmarked code over OTLP/HTTP to that receiver, so that CodSpeed can show the telemetry of each benchmark. The core sets up the export when it is imported, before the benchmarked modules get their tracers and loggers. It registers global tracer and logger providers and an `AsyncLocalStorage` context manager, which keeps the active span across async calls. It loads the OpenTelemetry SDK only when the endpoint is set, so the other modes never load it. If the benchmarked code already registered its own providers, the core keeps them and warns that their telemetry is not exported. Each span and log record also carries its times on the clock of the benchmark markers (`codspeed.monotonic_start_ns`, `codspeed.monotonic_end_ns`, `codspeed.monotonic_time_ns`). Trace and span ids come from `crypto`, because the benchmarks run with a seeded `Math.random()`. The mode runs each benchmark once, like simulation, but with the default V8 flags, so the timings stay close to production. The tinybench, benchmark.js and Vitest plugins wrap each benchmarked call with `withRoundMarkers`, which sends the start and end of the call as a round of the benchmark. The runner uses these rounds to find the benchmark of each span and log. The plugins also flush the export once their benchmarks are done, since Vitest may kill its worker before the process exits. With Vitest, a setup file imports the core in each worker, ahead of the benchmark files. Add the `examples/with-telemetry` project, which runs the same workloads with each of the three plugins. Closes COD-4149 Co-Authored-By: Claude Opus 5.5 <[email protected]>
…runs `withRoundMarkers` only waited for native promises. A benchmark returning another thenable, such as a database query, got its end marker as soon as it returned, so the spans and logs it emitted afterwards fell outside its round. It now waits for any object with a `then` method, and returns a native promise, which the plugins only await. The Vitest runners skipped `flushTelemetry()` when a benchmark or a hook threw, so Vitest could stop the worker with spans and logs still queued, including those that explain the failure. They now flush in a `finally` block. Refs COD-4149 Co-Authored-By: Claude Opus 5.5 <[email protected]>
The integration test only searched the exported protobuf bytes for names and attribute keys: it would pass with zero timestamps, unlinked spans or a log missing its span. Split the building of the providers out of `registerProviders` into `buildProviders`, which takes the span and log exporters, and test it with the SDK's in-memory exporters. The test checks that the child span links to its parent, that a log links to its span, and that the monotonic times of the spans and logs are in order. The subprocess test still covers the OTLP/HTTP export. Refs COD-4149 Co-Authored-By: Claude Opus 5.5 <[email protected]>
a81d3d5 to
5fc9260
Compare
| "@opentelemetry/context-async-hooks": "^3.1.0", | ||
| "@opentelemetry/exporter-logs-otlp-proto": "^0.301.0", | ||
| "@opentelemetry/exporter-trace-otlp-proto": "^0.301.0", | ||
| "@opentelemetry/resources": "^3.1.0", | ||
| "@opentelemetry/sdk-logs": "^3.1.0", | ||
| "@opentelemetry/sdk-trace": "^3.1.0", |
There was a problem hiding this comment.
Supported Node.js installs fail
The new OpenTelemetry SDK packages require Node.js >=22.15.0, but this repository still supports earlier Node.js 22 releases. If a user's package manager enforces dependency engine requirements, these regular dependencies prevent them from installing any CodSpeed plugin—even when they never enable telemetry. Use compatible dependencies or explicitly raise the supported minimum.
Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/core/package.json
Line: 44-49
Comment:
**Supported Node.js installs fail**
The new OpenTelemetry SDK packages require Node.js `>=22.15.0`, but this repository still supports earlier Node.js 22 releases. If a user's package manager enforces dependency engine requirements, these regular dependencies prevent them from installing any CodSpeed plugin—even when they never enable telemetry. Use compatible dependencies or explicitly raise the supported minimum.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Merging this PR will degrade performance by 11.51%4 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
Adds a
telemetryrunner mode, in which the core exports the OpenTelemetry spans and logs of the benchmarked code over OTLP/HTTP to the receiver atCODSPEED_OTEL_ENDPOINT. CodSpeed can then show the telemetry of each benchmark, as pytest-codspeed and codspeed-rust already do.How it works
@codspeed/coresets up the export when it is imported, before the benchmarked modules get their tracers and loggers. It registers global tracer and logger providers, and anAsyncLocalStoragecontext manager that keeps the active span acrossawaits. It loads the OpenTelemetry SDK withrequireonly when the endpoint is set, so the other modes never load it.codspeed.monotonic_start_ns/codspeed.monotonic_end_nson spans,codspeed.monotonic_time_nson logs.withRoundMarkers, which sends the start and end of the call as a round of the benchmark. The runner uses these rounds to find the benchmark of each span and log.Things worth a look
crypto: the benchmarks run with a seededMath.random(), so the SDK's default generator would give every process the same ids.@opentelemetry/*packages are regular dependencies of@codspeed/core. Making them optional peer dependencies would keep them out of the default install, at the cost of an install step for telemetry users.@opentelemetry/apiin particular is usually a peer dependency, so that the benchmarked code and the core share one copy.code.file.path/code.line.number, so the spans and logs carry no source location yet, unlike the Rust and Python integrations. Adding it means reading the call stack (util.getCallSites()) on each span and log; that can come in a follow-up.Example
examples/with-telemetryruns the same workloads (nested spans with logs, a log outside any span, a log through@opentelemetry/api-logs, and no telemetry) with each of the three plugins:pnpm bench-vitest,pnpm bench-tinybench,pnpm bench-benchmark-js.packages/core/tests/telemetry.integ.test.tsruns a script against a local OTLP receiver, and checks the exported spans and logs and their monotonic attributes. It also checks that the SDK isn't loaded without an endpoint.Closes COD-4149