Valid HTTP/1.1 request causes 503 Backend fetch failed

[DemiMarie]

Expected Behavior

Varnish should either ignore or process HTTP/1.1 and HTTP/2.0 trailers, as allowed by the standards.

Current Behavior

If a request with trailers arrives over HTTP/1.1, Varnish returns a 503 Backend fetch failed to the client. If the request with trailers instead arrives over HTTP/2, Varnish does something that causes HAProxy to return a 502 Bad Gateway error.

Possible Solution

Varnish should read and discard any trailers it finds.

Steps to Reproduce (for bugs)

1. Send a request with HTTP/1.1 or HTTP/2.0 trailers.
2. Observe that Varnish doesn’t handle them properly, and furthermore returns a wrong error message.

Context

I found this bug by looking at other PRs and confirmed it with my own testing. I’ve been testing various HTTP implementations and found that most of them have some sort of protocol non-conformance. Notably, Varnish does strictly validate field names (which NGINX does not) and values (which neither HAProxy nor NGINX do), and that’s a good thing.

Varnish Cache version

varnishd (varnish-7.7.0 revision bb7a918) Copyright (c) 2006 Verdens Gang AS Copyright (c) 2006-2025 Varnish Software Copyright 2010-2025 UPLEX - Nils Goroll Systemoptimierung

Operating system

Arch

Source of binary packages used (if any)

Arch extra repository

[dridi]

We have #4125 in progress currently postponed for other HTTP/2 changes. Since you are one of the rare trailers users, would you like to give it a try once ready? We currently have no ETA though.

Out of curiosity, is your use case gRPC or something else?

[DemiMarie]

I’m sending deliberately weird requests (both valid and invalid) to various HTTP implementations to see how they react. So far I have found:

• NGINX and HAProxy allow control characters in field values (though not CR, LF, or NUL).
• NGINX allows forbidden header and trailer names in HTTP/2+.
• HAProxy allows some forbidden header and trailer names in HTTP/2+.
• Varnish doesn’t check seem to check for forbidden header names in HTTP/2.
• NGINX drops HTTP/1.x trailers in requests, and does not check them for even basic syntactic validity. I have not tested HTTP/2 trailers. Trailers in responses can be passed through or dropped.
• HAProxy always keeps trailers.
• Varnish can’t handle trailers at all and fails with a misleading error message and nothing useful in the logs.
• Go, Hyper, HAProxy, and NGINX allow leading and/or trailing whitespace in HTTP/2+ field values. I can use this to make an upstream and downstream disagree on the exact value of a field value.
• NGINX allows HTTP/1.x field values to start or end with a tab.
• At least Go, HAProxy, and NGINX will allow HTTP/1.1 chunk extensions to contain almost anything that is not a newline, including (for at least HAProxy and NGINX) NUL bytes.
• nghttp3 drops the entire field if it has a bad value and is not a pseudo-header.
• Google’s Quiche accepts malformed HTTP/3 headers.

HTTP really needs a conformance test suite.

[dridi]

In that case it would be interesting to learn more about the offending header names we allow for h2.

Could you perform your trailer tests on #4125? It's not ready to merge, but it would at least be interesting to see how it fares in its current shape with your conformance endeavor.

[DemiMarie]

transfer-encoding (even if not chunked!), upgrade, connection, keep-alive, and proxy-connection are all accepted. The last is even passed to the backend.