transport: reduce garbage allocations in server path #3305
Description
I have a production go-grpc server workload that does near-zero heap allocation in the request handler. As I dial up qps, p99 client response times line up with gc duration on the server. The alloc_space profile largely points to http2 meta frame parsing as well as http2Server.operateHeaders() and it's descendants.
Fetching profile over HTTP from http://localhost:8081/debug/pprof/heap
Saved profile in /home/cy/pprof/pprof.quota_server.alloc_objects.alloc_space.inuse_objects.inuse_space.021.pb.gz
File: quota_server
Type: alloc_space
Time: Jan 9, 2020 at 12:13pm (PST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) web
(pprof) top
Showing nodes accounting for 25.35TB, 68.83% of 36.83TB total
Dropped 408 nodes (cum <= 0.18TB)
Showing top 10 nodes out of 68
flat flat% sum% cum cum%
8.95TB 24.30% 24.30% 8.95TB 24.30% golang.org/x/net/http2.(*Framer).readMetaFrame.func1
3.90TB 10.59% 34.90% 12.73TB 34.58% google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders
3.33TB 9.04% 43.93% 3.33TB 9.04% google.golang.org/grpc/internal/transport.(*decodeState).addMetadata
2.06TB 5.58% 49.51% 2.06TB 5.58% google.golang.org/grpc/internal/transport.(*itemList).enqueue
1.38TB 3.74% 53.25% 1.38TB 3.74% google.golang.org/grpc/internal/transport.newRecvBuffer
1.26TB 3.43% 56.68% 1.49TB 4.05% google.golang.org/grpc/internal/transport.(*http2Server).writeHeaderLocked
1.26TB 3.43% 60.11% 1.95TB 5.30% google.golang.org/grpc/internal/transport.(*http2Server).WriteStatus
1.15TB 3.11% 63.22% 4.24TB 11.52% slack/gen-src/proto_idl/quotaservice._Quota_Ratelimit_Handler
1.03TB 2.80% 66.03% 1.03TB 2.80% context.WithValue
1.03TB 2.80% 68.83% 1.03TB 2.80% google.golang.org/grpc/internal/transport.newWriteQuota
Also see attached image. This could likely be reproduced by writing a 'trivial' server implementation and sending a high request rate to it. Happy to provide more debug info.
