Skip to content

fix(lambda-events): make CloudWatch Logs AwsLogs round-trip and work with from_value - #1189

Open
0z5a wants to merge 1 commit into
aws:mainfrom
0z5a:fix/cloudwatch-logs-awslogs-round-trip
Open

0z5a wants to merge 1 commit into
aws:mainfrom
0z5a:fix/cloudwatch-logs-awslogs-round-trip

Conversation

@0z5a

@0z5a 0z5a commented Nov 15, 2026

Copy link
Copy Markdown

📬 Issue #, if available:

N/A

✍️ Description of changes:

cloudwatch_logs::AwsLogs (the awslogs field of LogsEvent) has hand-written Serialize/Deserialize impls, and they have two problems.

1. Serialized data is unpadded base64 that the crate's own deserializer rejects. Serialize encodes with STANDARD_NO_PAD, but Deserialize decodes with STANDARD. Since base64 0.20, STANDARD requires canonical padding (base64 0.13's decoder ignored padding; the crate moved from 0.13 to 0.21 in #650). So serde_json::to_string(&event) followed by serde_json::from_str::<LogsEvent> fails with Invalid padding whenever the gzip output is not a multiple of 3 bytes long, which is about two payloads in three. The two existing round-trip tests only pass because both of their payloads happen to compress to a multiple of 3 bytes (re-serialized data lengths 276 and 496). Other consumers of this format reject the unpadded output too: Go's base64.StdEncoding (which aws-lambda-go's CloudwatchLogsRawData.Parse uses) fails with illegal base64 data at input byte 280, and Python's base64.b64decode fails with Incorrect padding. The incoming payloads are padded. This crate's own example-cloudwatch_logs-event.json fixture ends in ==, and so does aws-lambda-go's cloudwatch-logs-event.json testdata. The captured event in test_deserialize_example ends in =. RFC 4648 §3.2 also requires padding unless the referring spec says otherwise.

2. Deserialize only works with deserializers that can lend borrowed strings. The visitor reads the map key as &str (map.next_key()? matched against "data"). serde_json::from_slice/from_str, which lambda_runtime uses, can lend borrowed strings, so direct invocations work. serde_json::from_value and serde_json::from_reader cannot, and both fail with invalid type: string "data", expected a borrowed string. The other hand-written visitors in the crate (custom_serde/headers.rs and iam) read their keys as Cow<str>, which works with both APIs; AwsLogs is the only one that borrows its map key. This breaks:

  • handlers that take LambdaEvent<serde_json::Value> and pick the concrete type with serde_json::from_value, a common pattern when one function is subscribed to several sources;
  • tests that build an event with serde_json::json! + from_value;
  • tests that load a saved sample event with from_reader.

Changes:

  • serialize data with STANDARD, so the output is padded;
  • read the map key through a small #[serde(field_identifier)] enum instead of &str. It accepts borrowed and owned keys without allocating, and the error messages are unchanged (unknown field `datax`, expected `data` and missing field `data` ). In a release-mode timing of 100k from_slice calls on the fixture (11 interleaved runs, best of 5 each), the median was 4779 ns per event on main and 4722 ns with this change. Reading the key as an owned String instead measured 4823 ns, which is why I used the enum.

Tests:

  • example_cloudwatch_logs_event_from_value_and_reader: deserializes the existing fixture through from_value and from_reader and compares the results with from_slice.
  • test_serialize_round_trip_pads_base64: serializes events of growing message length, checks that each one deserializes back to an equal value and that data decodes with STANDARD. It also asserts that at least one payload length was not a multiple of 3, so the test cannot pass by luck.

Both tests fail on main:

test event::cloudwatch_logs::test::example_cloudwatch_logs_event_from_value_and_reader ... FAILED
test event::cloudwatch_logs::test::example_cloudwatch_logs_event ... ok
test event::cloudwatch_logs::test::test_deserialize_example ... ok
test event::cloudwatch_logs::test::test_serialize_round_trip_pads_base64 ... FAILED
called `Result::unwrap()` on an `Err` value: Error("invalid type: string \"data\", expected a borrowed string", line: 0, column: 0)
called `Result::unwrap()` on an `Err` value: Error("Invalid padding", line: 1, column: 304)

With this change both tests pass:

test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 233 filtered out; finished in 0.01s

I ran these commands on macOS arm64 (stable Rust, nightly rustfmt):

  • cargo test -p aws_lambda_events --all-features
  • cargo test -p aws_lambda_events
  • cargo test -p aws_lambda_events --no-default-features --features cloudwatch_logs
  • cargo +nightly fmt --all -- --check
  • cargo clippy -p aws_lambda_events --all-features -- -D warnings

All of them pass.

🔏 By submitting this pull request

  • I confirm that I've ran cargo +nightly fmt.
  • I confirm that I've ran cargo clippy --fix.
  • I confirm that I've made a best effort attempt to update all relevant documentation.
  • I confirm that my contribution is made under the terms of the Apache 2.0 license.

This change was drafted with Claude Code (claude-opus-5-5); the test results above are from the runs described.

…k with `from_value`

Serialize `data` with padded STANDARD base64. Deserialize decodes with STANDARD, which requires canonical padding since base64 0.20, so the unpadded output failed to deserialize for about two payloads in three.

Read the map key through a field-identifier enum instead of `&str`, so deserializers that cannot lend borrowed strings (`serde_json::from_value`, `from_reader`) work too.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant