feat: add auto_encoding and improve send_compressed behavior#2058
Closed
m62624 wants to merge 10 commits intohyperium:masterfrom
Closed
feat: add auto_encoding and improve send_compressed behavior#2058m62624 wants to merge 10 commits intohyperium:masterfrom
auto_encoding and improve send_compressed behavior#2058m62624 wants to merge 10 commits intohyperium:masterfrom
Conversation
…vior
The service was initially configured to send compressed data only in Gzip format. Different clients requested varying compression algorithms (e.g., Gzip and Zstd).
Logs showed the following behavior:
1. Client 1 requested Gzip:
```
Server received request: data="gzip", headers=MetadataMap { headers: {"te": "trailers", "content-type": "application/grpc", "grpc-encoding": "gzip", "grpc-accept-encoding": "gzip,identity", "user-agent": "tonic/0.13.0"} } Server sending response: gzip Responding with gzip compression
```
2. Client 2 requested Zstd:
```
Server received request: data="zstd", headers=MetadataMap { headers: {"te": "trailers", "content-type": "application/grpc", "grpc-encoding": "gzip", "grpc-accept-encoding": "zstd,identity", "user-agent": "tonic/0.13.0"} }
```
It is clear that clients requested different encoders using `grpc-accept-encoding`.
**Expected behavior:**
- The server should select a compression algorithm from the intersection of the supported algorithms from the server and the client.
- If the client requests a compression algorithm not supported by the server, the server should either use `identity` (no compression) or return a `UNIMPLEMENTED` status.
**Actual behavior:**
If any `send_compressed` configuration was set for the service, the `from_accept_encoding_header` method was triggered regardless of the `grpc-accept-encoding` header provided by the client.
The function responsible for compression selection on the server did not respect `send_compression_encodings`.
The server defaulted to the first supported compression algorithm from `grpc-accept-encoding` without checking if the algorithm was allowed by `send_compression_encodings`.
**Resolution:**
- If `send_compressed` is not specified, the server defaults to `identity`.
- If `send_compressed` is specified, the server only compresses responses for clients explicitly requesting a supported algorithm.
**Feature:**
The `auto_encoding` mechanism was added. This allows the server to automatically
select a compression algorithm supported by the client without requiring explicit configuration of `send_compressed` on the server.
The server dynamically adjusts compression behavior, ensuring compatibility with client preferences for efficient communication.
|
Did you use AI to generate the description? If so, could you please share it? |
Author
Starting from the "New Test Suite:" section, I generated the content using GitHub Copilot directly on the PR submission page. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
name: 💡 Feature Request
about: feat: Add
auto_encodingand improvesend_compressedbehaviorFeature Request
Crates
tonicMotivation
When using
send_compressed, the server sends compressed data in a format that may not be explicitly allowed for the service, even if the format isn't listed as supported by the server. This means that the client receives the desired compression result, even if the server has not explicitly enabled that compression algorithm for the service.The server should only send compressed data using algorithms that are supported by both the client and the server. If the client requests a compression format that the server does not support, the server should either send the data uncompressed (using
identity) or return an error (Status).Currently:
send_compressedis set for the service, thefrom_accept_encoding_headermethod executes regardless of the providedgrpc-accept-encoding.send_compression_encodingsconfiguration.This means that when processing a response, the server uses the
grpc-accept-encodingheader sent by the client and chooses the first supported compression algorithm, ignoring whether that algorithm is explicitly allowed insend_compression_encodings.This results in unexpected behavior, as the server should only use compression types explicitly allowed via
send_compressed.Proposal
The logic has been refactored to address this issue:
send_compressedis used and the server-client compression formats do not overlap, the server sends data without compression (identity).auto_encodingmethod has been added to automatically select the correct compression format based on the intersection of client and server-supported formats. That is, the server sends the desired format to each client.This change ensures alignment with expected gRPC behavior.
Alternatives
Drawback: It continues to cause unexpected issues when formats don't align.
Drawback: It might unnecessarily interrupt communication where uncompressed data could suffice.
New Test Suite:
Cargo.toml: Added a new test directorytests/various_compression_formatsto the members list.tests/various_compression_formats/Cargo.toml: Created a new Cargo.toml file for thevarious_compression_formatstest suite.tests/various_compression_formats/build.rs: Added a build script to compile protobuf definitions.tests/various_compression_formats/proto/proto_box.proto: Defined protobuf messages and services for the test suite.tests/various_compression_formats/src/lib.rs: Included the compiled protobuf module.tests/various_compression_formats/tests/auto_encoding.rs: Added comprehensive tests to verify the behavior of different compression formats and automatic encoding detection.Automatic Encoding Feature:
tonic-build/src/server.rs: Introduced theauto_encodingflag and logic to automatically determine the encoding based on request headers. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]tonic/src/codec/compression.rs: Modified thefrom_accept_encoding_headerfunction to support theauto_encodingflag.tonic/src/server/grpc.rs: Added theauto_encodingflag to theGrpcstruct and implemented the logic to handle automatic encoding. [1] [2] [3] [4]