check issue with nginx abour chunked header

[qizhendong1]

nginx/nginx#316

[longmaythesunshine-zx]

The Transfer-Encoding header is a hop-by-hop header, therefore it is not a bug if the upstreams don't receive this header

[qizhendong1]

The Transfer-Encoding header is a hop-by-hop header, therefore it is not a bug if the upstreams don't receive this header

Does this mean that even if the client carries the Transfer-Encoding request header, the upsteam may not receive it?

This is fine in some cases, but when the client uses the Transfer-Encoding request header as a signature field, the request received by the upstream does not have this request header, which means the signature does not match.

I am not sure whether the connection between nginx and the upstream in this request also uses Transfer-Encoding ?

[echo-97]

Upon reviewing the Nginx code, I found that when the client uses Transfer-Encoding, Nginx will use Transfer-Encoding: chunked for communication with the client. However, when communicating with the upstream server, it defaults to a non-chunked transfer when there is a request body. If there is no request body, it does not process the Transfer-Encoding header and forwards it to the server as is

[qizhendong1]

Upon reviewing the Nginx code, I found that when the client uses Transfer-Encoding, Nginx will use Transfer-Encoding: chunked for communication with the client. However, when communicating with the upstream server, it defaults to a non-chunked transfer when there is a request body. If there is no request body, it does not process the Transfer-Encoding header and forwards it to the server as is

Your opinion is consistent with the results of my test.

I think this is an issue because upstreams need to know whether the client passes the Transfer-Encoding request header in some specific cases. We have encountered such a problem in the production environment.

[qizhendong1]

The nginx community has not fixed this problem. Is it because nginx cannot proxy requests in the chunked way during the interaction with the upstream? For example, nginx cannot ensure that the size of each chunk on the client is consistent with the size of the proxy to the upstream.

[echo-97]

You can try uploading large files; I found in the Nginx code that when uploading large files, it is possible to send them in chunked format to the upstream server.

[qizhendong1]

I test that nginx proxy with Transfer-Encoding header when client(client always with Transfer-Encoding header) body is >1024 bytes and body is empty to upstream.

It agrees with what you said.

But I don't find the code, can you tell me ? @echo-97

[qizhendong1]

Besides that, I found a bug in the python requests library when set Transfer-Encoding header

[echo-97]

Okay, in the function ngx_http_read_client_request_body, when the request body has not been completely received and the current TCP connection is idle, it is possible to set r->reading_body to 1. In the function ngx_http_proxy_create_request:
ngx_http_proxy_create_request() {
if (plcf->body_lengths) {
// ignore
} else if (r->headers_in.chunked && r->reading_body) {
ctx->internal_body_length = -1;
ctx->internal_chunked = 1;
} else {
// ignore
}

// ignore

if (r->request_body_no_buffering) {
    u->request_bufs = cl;

    if (ctx->internal_chunked) {
        u->output.output_filter = ngx_http_proxy_body_output_filter;
        u->output.filter_ctx = r;
    }
    // ignore
}

}
When sending data to the upstream, the function ngx_http_proxy_body_output_filter is called, which constructs chunked data.

[qizhendong1]

I modified the code and simply tested it and it works

diff --git a/src/http/modules/ngx_http_proxy_module.c b/src/http/modules/ngx_http_proxy_module.c
index b774c866e..fb2ba294c 100644
--- a/src/http/modules/ngx_http_proxy_module.c
+++ b/src/http/modules/ngx_http_proxy_module.c
@@ -1342,6 +1342,9 @@ ngx_http_proxy_create_request(ngx_http_request_t *r)
         ctx->internal_chunked = 1;

     } else {
+        if(r->headers_in.chunked) {
+            ctx->internal_chunked = 1;
+        }
         ctx->internal_body_length = r->headers_in.content_length_n;
     }