Skip to content

Drop a wildcard from a filtered Accept-Encoding - #2392

Open
pavel-ptashyts wants to merge 1 commit into
AsyncHttpClient:mainfrom
maygemdev:fix/accept-encoding-wildcard
Open

pavel-ptashyts wants to merge 1 commit into
AsyncHttpClient:mainfrom
maygemdev:fix/accept-encoding-wildcard

Conversation

@pavel-ptashyts

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #2381, as asked in its review. With automatic decompression on and the Brotli or Zstd library missing, a * in a caller-set Accept-Encoding still offered the missing coding:

  • gzip, * and * went out unchanged, and the * covers both br and zstd;
  • br, * with both libraries missing went out as br;q=0, *, and the * still covers zstd.

A server that picks the missing coding sends a body AHC cannot decode, and the handler receives it still compressed.

Netty interop

The review on #2381 noted that Netty's HttpContentCompressor picks br for gzip, br;q=0, *. I checked determineEncoding in Netty 4.2.20 (the version in pom.xml) and ran it in process in the new tests. It matches * and coding names by substring (x-zstd counts as zstd, x* as *). When any listed coding has a positive weight it walks br, zstd, snappy, gzip, deflate and takes the first listed one whose weight is at least the next one's, q=0 included; only otherwise does it read the *. So:

  • appending br;q=0, zstd;q=0 next to the * would make gzip, * get br from a Netty server where main gets gzip;
  • replacing the * with br or zstd would make gzip, * get them over HTTP/2, where Http2ContentDecompressor decodes only gzip and deflate;
  • dropping or adding any other element can move its pick along that chain.

Change

HttpUtils.filterOutAcceptEncoding, used by filterOutBrotliFromAcceptEncoding / filterOutZstdFromAcceptEncoding, now handles a list with an accepting * in a new private replaceWildcard:

  • The * is dropped, with every element naming the missing coding, every other token holding a *, and every weight 0 refusal but identity;q=0 and *;q=0. With no * left, a coding the list does not name is not acceptable (RFC 9110 section 12.5.3), so the refusals add nothing; identity;q=0 and *;q=0 still decide whether an unencoded response is acceptable.
  • The elements with a positive weight stay as written.
  • Only when no content coding but br or zstd is left with a positive weight, gzip and deflate take the place of the *, at its weight, unless the list names them (x-gzip counts as gzip, RFC 9110 section 8.4.1.3; x-deflate as deflate). Both are decoded over HTTP/1.1 and HTTP/2.

So Netty's pick on the filtered list is the pick it makes on main's list, unless that pick was the missing coding, a refused coding or one only the * offered; then it moves to the next positive coding the caller listed, to gzip or deflate, or to no coding.

Examples with both libraries missing: * -> gzip, deflate; br, * -> gzip, deflate; gzip, * -> gzip; gzip, br, * -> gzip (was gzip, br;q=0, *, which got br from Netty); *;q=0.5 -> gzip;q=0.5, deflate;q=0.5; identity;q=0, * -> identity;q=0, gzip, deflate.

Without an accepting * the filter does what #2381 made it do. No public API change; with both libraries present, or automatic decompression off, the header is left as the caller set it.

Tests

  • HttpUtilsTest: the * replaced by gzip and deflate, at its weight, unless named or refused (also via x-gzip / x-deflate); dropped next to another offered coding; refusals and *-holding tokens dropped; identity and *;q=0 kept; refusals still kept without a *.
  • NettyRequestFactoryAcceptEncodingTest: newNettyRequest with the availability flags, automatic decompression on and off; and a real HttpContentCompressor in an EmbeddedChannel that answers each header AHC sends, for each flag combination with a library missing, asserting it picks a coding AHC decodes and not one the caller refused, and that gzip, * still gets gzip, which also covers HTTP/2.
  • On main 6 of these tests fail, for example * with Brotli missing gets br from the Netty server, and gzip, br, * gets br.

Not committed: a sweep over 7.1 million generated lists (up to four elements from 37 forms, including the substring cases above), for each combination with a library missing, running both main's filter and this one through Netty's determineEncoding. It found no list where this change gives a coding AHC cannot decode, over HTTP/1.1 or HTTP/2, while main gives a decodable one; it found about 358,000 where main gives an undecodable one over HTTP/1.1 and this change does not. Run on the versions reviewed in passes 4 and 5 below, it finds the regressions reported there.

mvnw.cmd clean verify passes locally on JDK 17 (no JDK 11 on the host): 2069 tests, 0 failures, 33 skipped, Revapi green. CI covers JDK 11.

Not changed

  • Without a *, a refusal of the missing coding is still sent: with Brotli missing, gzip, br;q=0 gets br from a Netty server, as on main and as kept in Drop undecodable codings from Accept-Encoding #2381. Dropping it is safe only when no token holds a *: br;q=0, x* would otherwise get br, where main gets zstd. Left for a separate change.
  • A * next to another offered coding is dropped without adding gzip or deflate, so compress, * becomes compress, and a server that would have used gzip sends no coding. Adding them there can move Netty's pick to a coding HTTP/2 cannot decode.
  • With both libraries present, a * still offers every coding, also ones AHC cannot decode, such as compress. The filter runs only when a library is missing.
  • HTTP/2: Http2ContentDecompressor decodes only gzip and deflate, so a caller's explicit br or zstd over h2 needs its own issue, as noted on Drop undecodable codings from Accept-Encoding #2381.

Second-agent review

Two agent reviewers, each a fresh read-only CLI run on git diff upstream/main...HEAD, reasoning effort medium; the model and effort below are as reported by each run. Both were asked to check the result against Netty 4.2.20's determineEncoding over HTTP/1.1 and HTTP/2.

Pass Commit Guided: Codex gpt-6-astra Independent: Claude claude-opus-5-5
1 77a9e4dea With Brotli missing, x-br;q=0, ZSTD;q=0, * got br from Netty (fixed in cb588ef5b) br;q=0, * with zstd missing got the refused br (fixed in cb588ef5b)
2 cb588ef5b Dropping br;q=0 without a * made br;q=0, x* get br (fixed in 70d19d6b9) No actionable findings
3 70d19d6b9 x* left next to the replaced * made Netty pick br (fixed in a5cbba837) Adding br or zstd in place of the * made gzip, * get them over HTTP/2 (fixed in a5cbba837)
4 a5cbba837 Dropping an unknown x-snappy;q=0.2 moved Netty's pick to zstd over HTTP/2 (fixed in bb85447b6) No regression; gzip, br;q=0 without a * still gets br (not changed, see "Not changed")
5 bb85447b6 Keeping x-zstd;q=0 while dropping br;q=0 made Netty pick zstd (fixed in 649b28223) No actionable findings; suggested dropping every refusal (done in 649b28223)
6 649b28223 No code findings; one commit summary was 51 characters (the branch is squashed into e98cbfc, same tree as 649b28223) No regression; gzip, br;q=0 without a * again (not changed, see "Not changed")

Optional notes left as they are:

  • Parameters other than q on the * are copied to gzip and deflate (*;foo=bar -> gzip;foo=bar, deflate;foo=bar); RFC 9110 allows only a weight there, and Netty reads such input as weight 0 before and after.
  • A list that refuses everything but the *, such as identity;q=0, gzip;q=0, deflate;q=0, *, becomes identity;q=0, which a strict server may answer with 406. That is what the list means once the missing coding is not offered.
  • Codex noted on each pass that the AGENTS.md gate asks for clean verify on JDK 11; locally it ran on JDK 17, and CI covers JDK 11.

Claude Code on behalf of Pavel Ptashyts

🤖 Generated with Claude Code

With automatic decompression on and Brotli or Zstd missing, a "*" in
a caller-set Accept-Encoding still offered the missing coding:
"gzip, *" went out unchanged and "br, *" as "br;q=0, *". A server
could then answer with a body AHC cannot decode.

Refusing the coding next to the "*" is not enough, and neither is
replacing the "*" with every coding AHC decodes. Netty's
HttpContentCompressor matches "*" and coding names by substring,
picks a refused coding once another listed coding has a positive
weight, and compares weights along br, zstd, snappy, gzip, deflate,
so "gzip, br;q=0, *" gets br, and adding br or zstd would make
"gzip, *" get a coding HTTP/2 does not decode.

The "*" is now dropped, with the elements naming the missing coding,
other tokens holding a "*", and refusals but "identity;q=0" and
"*;q=0", which add nothing once no "*" is left (RFC 9110 section
12.5.3). Elements with a positive weight stay as written. Only when
no coding but br or zstd is left offered, gzip and deflate, which
both HTTP/1.1 and HTTP/2 decode, take the place of the "*". Netty
then picks what it picks on main, unless that was the missing coding,
a refused coding or one only the "*" offered. A test runs the result
through a real HttpContentCompressor.

Follow-up to AsyncHttpClient#2381.

Claude Code on behalf of Pavel Ptashyts

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@pavel-ptashyts

Copy link
Copy Markdown
Contributor Author

@hyperxpro The one failing check, test (ubuntu-latest, 11), failed before any test ran: Maven could not download org.apache.maven.surefire:surefire-junit-platform:jar:3.6.1 from Maven Central on that runner ("The following artifacts could not be resolved ... Could not find artifact ... in central"), so the job stopped after 33 s. The same artifact resolved in the other 12 jobs, which all passed, including Windows and macOS on JDK 11, and compile-and-check passed too. It looks like a transient Maven Central or network issue, not this change.

I can't re-run jobs on this repository. Could you re-run the failed job if you need it green? Thanks.

Claude Code on behalf of Pavel Ptashyts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant