.NET servers reject https gRPC requests terminated to plaintext by Envoy

[jukie]

Description:
This appears to be the same behavior reported in istio/istio#34448 and has to do with the way Envoy preserves :scheme.

When an application makes an https gRPC request to a .NET server behind an Istio proxy or gateway and Envoy terminates the request to plaintext, starting in 1.10 Envoy preserves :scheme as https. .NET sees the mismatch between the scheme and the fact that the request from Envoy is plaintext and raises an error, rejecting the request.

My org runs a lot of .NET services and during the migration from ingress-nginx this has become an issue for grpc services. I think we should expose the ability to configure SchemeHeaderTransformation in BackendTrafficPolicy so that users can configure this. Since SchemeHeaderTransformation is an HCM level configuration ClientTrafficPolicy is right place to add this.

Repro steps:

• Build a .NET backend with a gRPC listenter
• GRPCRoute with TLS termination at the gateway and plaintext upstream

[heggi]

Workaround for .Net: disable :scheme header check.
https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.server.kestrel.core.kestrelserveroptions.allowalternateschemes

[jukie]

Thanks! Yep there's a viable workaround in the kestrel config but making this configurable in the gateway layer still seems reasonable to me. Asking people migrating from ingress-nginx to make code changes isn't a great experience.

[heggi]

Yep, fixing the gateway is a right way, but it may take a lot of time. A workaround may be required for it to work at this moment.

[jukie]

I've tested the changes from my PR with a .NET gRPC service and that should be all we need. If we can get this into EG v1.7 that would fix it for everyone.