ObjectDisposedException during HTTP/2 mTLS handshake in production Kubernetes environment

[ranavivek04]

ObjectDisposedException during HTTP/2 mTLS handshake in Kubernetes production environment

What version of gRPC and what language are you using?

Language: C# / .NET 8.0
gRPC Packages:

• Grpc.AspNetCore: 2.71.0
• Grpc.Net.Client: 2.71.0 (implicit via Grpc.AspNetCore)
• Grpc.Tools: 2.72.0
• Google.Protobuf: 3.32.0

What operating system (Linux, Windows,...) and version?

Development Environment: Windows 10.0.26100 (Windows 11)
Production Environment: Linux containers in Kubernetes (Azure Kubernetes Service)

• Container Runtime: Linux containers
• Content root path: /usr/local/xyz/abc

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

.NET SDK: 8.0.413
Runtime: Microsoft.AspNetCore.App 8.0.19
Target Framework: net8.0

What did you do?

Scenario: Health probe service making mTLS gRPC calls to a TTS service in Kubernetes production environment.

Code Pattern:

// Create gRPC channel with mTLS client certificate
var clientCert = X509Certificate2.CreateFromPemFile(certPath, keyPath);
var httpHandler = new HttpClientHandler
{
    ClientCertificates = { clientCert }
};

using var channel = GrpcChannel.ForAddress(serviceUrl, new GrpcChannelOptions
{
    HttpHandler = httpHandler
});

var client = new Synthesizer.SynthesizerClient(channel);
var reply = client.Synthesize(request, headers);

await foreach (var message in reply.ResponseStream.ReadAllAsync())
{
    // Process response
}

Environment Details:

• Service URL: https://headless-service-url:8081
• mTLS with client certificates loaded from PEM files
• Kubernetes service-to-service communication
• Health check pattern with 600-second intervals

What did you expect to see?

Successful HTTP/2 gRPC connection establishment and streaming response processing without disposal errors.

What did you see instead?

Error: ObjectDisposedException: Cannot access a disposed object. Object name: 'System.Net.Security.SslStream'

Full Stack Trace:

Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error starting gRPC call. HttpRequestException: An error occurred while sending the request. IOException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake. ObjectDisposedException: Cannot access a disposed object. Object name: 'System.Net.Security.SslStream'.", DebugException="System.Net.Http.HttpRequestException: An error occurred while sending the request.")
---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake.
---> System.ObjectDisposedException: Cannot access a disposed object. Object name: 'System.Net.Security.SslStream'.
   at System.Net.Security.SslStream.<ThrowIfExceptional>g__ThrowExceptional|126_0(ExceptionDispatchInfo e)
   at System.Net.Security.SslStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Http.Http2Connection.SetupAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SetupAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConstructHttp2ConnectionAsync(Stream stream, HttpRequestMessage request, IPEndPoint remoteEndPoint, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnectionPool.ConstructHttp2ConnectionAsync(Stream stream, HttpRequestMessage request, IPEndPoint remoteEndPoint, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(QueueItem queueItem)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
   --- End of inner exception stack trace ---
   at Grpc.Net.Client.Internal.HttpContentClientStreamReader`2.MoveNextCore(CancellationToken cancellationToken)
   at Grpc.Core.AsyncStreamReaderExtensions.ReadAllAsyncCore[T](IAsyncStreamReader`1 streamReader, CancellationToken cancellationToken)+MoveNext()
   at Grpc.Core.AsyncStreamReaderExtensions.ReadAllAsyncCore[T](IAsyncStreamReader`1 streamReader, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()

Anything else we should know about your project / environment?

Key Observations:

1. Issue occurs consistently in Kubernetes production environment
2. Timing: ~16 seconds after application startup, during first gRPC call attempt
3. mTLS specific: Issue appears when using client certificates for mutual TLS
4. HTTP/2 related: Error occurs during HTTP/2 handshake setup phase

Workarounds Attempted:

• Fresh HttpClientHandler creation per call
• Connection pooling disabled (PooledConnectionLifetime = TimeSpan.Zero)
• OCSP/CRL check disabling
• Multiple timeout configurations
• DisposeHttpClient = true in GrpcChannelOptions
• Custom SSL certificate validation callbacks

Environment Context:

• Kubernetes service mesh: Complex network infrastructure
• SSL handshake timing: Often takes 15+ seconds in this environment
• Certificate management: PEM files loaded from mounted volumes

Similar Issues:
This appears related to existing issues around HTTP/2 connection establishment and SSL stream lifecycle management in containerized environments with high-latency SSL handshakes.

Reproduction:
Issue reproduces consistently in Kubernetes production environment but not in local development, suggesting environmental factors (network latency, SSL negotiation timing) contribute to the disposal race condition.

[ranavivek04]

@JamesNK it seems the issue is .NET gRPC Client TLS/HTTP2 Negotiation Failure with mTLS Server.
.NET gRPC client fails to establish mTLS connections that work perfectly with curl, indicating a .NET-specific TLS implementation issue.

Key Evidence:# Both TLS 1.2 and 1.3 work perfectly with curl
curl -k --cert client.crt --key client.key --tlsv1.2 https://server:8081
curl -k --cert client.crt --key client.key --tlsv1.3 https://server:8081

Questions:
TLS Protocol Negotiation: Why does .NET gRPC fail TLS negotiation when curl succeeds with identical certificates?

HTTP/2 ALPN: Are there known issues with HTTP/2 ALPN negotiation in mTLS scenarios?

Debugging: How can we get better visibility into the TLS handshake process within gRPC?

Lazy Connection: Is there a way to force immediate connection validation instead of lazy establishment?

[JamesNK]

Most HTTP/2 and TLS logic is in the HTTP handler. Talking with the HTTP handler owners like you are here - dotnet/runtime#119022 - is the right place to figure this out.

There is some connection management in Grpc.Net.Client to support load balancing, but that is only enabled when you're using SocketsHttpHandler, and your code sample shows you're using HttpClientHandler.

If you were using SocketsHttpHandler you could disable that connection management with:

        var handler = new SocketsHttpHandler();
        handler.Properties["__GrpcLoadBalancingDisabled"] = true;
        _channel = GrpcChannel.ForAddress(environment.Address, new GrpcChannelOptions
        {
            Credentials = ChannelCredentials.Create(new SslCredentials(), credentials),
            HttpHandler = handler
        });