Repository navigation
Drop a wildcard from a filtered Accept-Encoding - #2392
Open
pavel-ptashyts wants to merge 1 commit into
Open
pavel-ptashyts wants to merge 1 commit into
pavel-ptashyts wants to merge 1 commit into
Conversation
With automatic decompression on and Brotli or Zstd missing, a "*" in a caller-set Accept-Encoding still offered the missing coding: "gzip, *" went out unchanged and "br, *" as "br;q=0, *". A server could then answer with a body AHC cannot decode. Refusing the coding next to the "*" is not enough, and neither is replacing the "*" with every coding AHC decodes. Netty's HttpContentCompressor matches "*" and coding names by substring, picks a refused coding once another listed coding has a positive weight, and compares weights along br, zstd, snappy, gzip, deflate, so "gzip, br;q=0, *" gets br, and adding br or zstd would make "gzip, *" get a coding HTTP/2 does not decode. The "*" is now dropped, with the elements naming the missing coding, other tokens holding a "*", and refusals but "identity;q=0" and "*;q=0", which add nothing once no "*" is left (RFC 9110 section 12.5.3). Elements with a positive weight stay as written. Only when no coding but br or zstd is left offered, gzip and deflate, which both HTTP/1.1 and HTTP/2 decode, take the place of the "*". Netty then picks what it picks on main, unless that was the missing coding, a refused coding or one only the "*" offered. A test runs the result through a real HttpContentCompressor. Follow-up to AsyncHttpClient#2381. Claude Code on behalf of Pavel Ptashyts Co-Authored-By: Claude Opus 5.5 <[email protected]>
Contributor
Author
|
@hyperxpro The one failing check, I can't re-run jobs on this repository. Could you re-run the failed job if you need it green? Thanks. Claude Code on behalf of Pavel Ptashyts |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #2381, as asked in its review. With automatic decompression on and the Brotli or Zstd library missing, a
*in a caller-setAccept-Encodingstill offered the missing coding:gzip, *and*went out unchanged, and the*covers bothbrandzstd;br, *with both libraries missing went out asbr;q=0, *, and the*still coverszstd.A server that picks the missing coding sends a body AHC cannot decode, and the handler receives it still compressed.
Netty interop
The review on #2381 noted that Netty's
HttpContentCompressorpicksbrforgzip, br;q=0, *. I checkeddetermineEncodingin Netty 4.2.20 (the version inpom.xml) and ran it in process in the new tests. It matches*and coding names by substring (x-zstdcounts aszstd,x*as*). When any listed coding has a positive weight it walksbr,zstd,snappy,gzip,deflateand takes the first listed one whose weight is at least the next one's,q=0included; only otherwise does it read the*. So:br;q=0, zstd;q=0next to the*would makegzip, *getbrfrom a Netty server wheremaingetsgzip;*withbrorzstdwould makegzip, *get them over HTTP/2, whereHttp2ContentDecompressordecodes only gzip and deflate;Change
HttpUtils.filterOutAcceptEncoding, used byfilterOutBrotliFromAcceptEncoding/filterOutZstdFromAcceptEncoding, now handles a list with an accepting*in a new privatereplaceWildcard:*is dropped, with every element naming the missing coding, every other token holding a*, and every weight 0 refusal butidentity;q=0and*;q=0. With no*left, a coding the list does not name is not acceptable (RFC 9110 section 12.5.3), so the refusals add nothing;identity;q=0and*;q=0still decide whether an unencoded response is acceptable.brorzstdis left with a positive weight,gzipanddeflatetake the place of the*, at its weight, unless the list names them (x-gzipcounts asgzip, RFC 9110 section 8.4.1.3;x-deflateasdeflate). Both are decoded over HTTP/1.1 and HTTP/2.So Netty's pick on the filtered list is the pick it makes on
main's list, unless that pick was the missing coding, a refused coding or one only the*offered; then it moves to the next positive coding the caller listed, to gzip or deflate, or to no coding.Examples with both libraries missing:
*->gzip, deflate;br, *->gzip, deflate;gzip, *->gzip;gzip, br, *->gzip(wasgzip, br;q=0, *, which gotbrfrom Netty);*;q=0.5->gzip;q=0.5, deflate;q=0.5;identity;q=0, *->identity;q=0, gzip, deflate.Without an accepting
*the filter does what #2381 made it do. No public API change; with both libraries present, or automatic decompression off, the header is left as the caller set it.Tests
HttpUtilsTest: the*replaced by gzip and deflate, at its weight, unless named or refused (also viax-gzip/x-deflate); dropped next to another offered coding; refusals and*-holding tokens dropped;identityand*;q=0kept; refusals still kept without a*.NettyRequestFactoryAcceptEncodingTest:newNettyRequestwith the availability flags, automatic decompression on and off; and a realHttpContentCompressorin anEmbeddedChannelthat answers each header AHC sends, for each flag combination with a library missing, asserting it picks a coding AHC decodes and not one the caller refused, and thatgzip, *still getsgzip, which also covers HTTP/2.main6 of these tests fail, for example*with Brotli missing getsbrfrom the Netty server, andgzip, br, *getsbr.Not committed: a sweep over 7.1 million generated lists (up to four elements from 37 forms, including the substring cases above), for each combination with a library missing, running both
main's filter and this one through Netty'sdetermineEncoding. It found no list where this change gives a coding AHC cannot decode, over HTTP/1.1 or HTTP/2, whilemaingives a decodable one; it found about 358,000 wheremaingives an undecodable one over HTTP/1.1 and this change does not. Run on the versions reviewed in passes 4 and 5 below, it finds the regressions reported there.mvnw.cmd clean verifypasses locally on JDK 17 (no JDK 11 on the host): 2069 tests, 0 failures, 33 skipped, Revapi green. CI covers JDK 11.Not changed
*, a refusal of the missing coding is still sent: with Brotli missing,gzip, br;q=0getsbrfrom a Netty server, as onmainand as kept in Drop undecodable codings from Accept-Encoding #2381. Dropping it is safe only when no token holds a*:br;q=0, x*would otherwise getbr, wheremaingetszstd. Left for a separate change.*next to another offered coding is dropped without adding gzip or deflate, socompress, *becomescompress, and a server that would have used gzip sends no coding. Adding them there can move Netty's pick to a coding HTTP/2 cannot decode.*still offers every coding, also ones AHC cannot decode, such ascompress. The filter runs only when a library is missing.Http2ContentDecompressordecodes only gzip and deflate, so a caller's explicitbrorzstdover h2 needs its own issue, as noted on Drop undecodable codings from Accept-Encoding #2381.Second-agent review
Two agent reviewers, each a fresh read-only CLI run on
git diff upstream/main...HEAD, reasoning effort medium; the model and effort below are as reported by each run. Both were asked to check the result against Netty 4.2.20'sdetermineEncodingover HTTP/1.1 and HTTP/2.gpt-6-astraclaude-opus-5-5x-br;q=0, ZSTD;q=0, *gotbrfrom Netty (fixed in cb588ef5b)br;q=0, *with zstd missing got the refusedbr(fixed in cb588ef5b)br;q=0without a*madebr;q=0, x*getbr(fixed in 70d19d6b9)x*left next to the replaced*made Netty pickbr(fixed in a5cbba837)brorzstdin place of the*madegzip, *get them over HTTP/2 (fixed in a5cbba837)x-snappy;q=0.2moved Netty's pick tozstdover HTTP/2 (fixed in bb85447b6)gzip, br;q=0without a*still getsbr(not changed, see "Not changed")x-zstd;q=0while droppingbr;q=0made Netty pickzstd(fixed in 649b28223)gzip, br;q=0without a*again (not changed, see "Not changed")Optional notes left as they are:
qon the*are copied to gzip and deflate (*;foo=bar->gzip;foo=bar, deflate;foo=bar); RFC 9110 allows only a weight there, and Netty reads such input as weight 0 before and after.*, such asidentity;q=0, gzip;q=0, deflate;q=0, *, becomesidentity;q=0, which a strict server may answer with 406. That is what the list means once the missing coding is not offered.clean verifyon JDK 11; locally it ran on JDK 17, and CI covers JDK 11.Claude Code on behalf of Pavel Ptashyts
🤖 Generated with Claude Code