Mysterious simultaneous nil pointer dereference in client.go

[paul-at-cybr]

Hi!
We had a roughly 1-minute outage on vault-csi-provider, where the debug trail ends on internal/client/client.go:159. The issue was likely caused by some API consumed by vault-csi-provider, because all 9 pods had the same segfault at the same time.

This is vault-csi-provider v1.4.3 via the latest vault-helm release (v0.28.1)

Since I'm not an expert on Vault or K8S internals, I'm unable to draw any helpful conclusions regarding the root cause - The best I can do is provide some lightly obfuscated logs.

[INFO]  server: Processing unary gRPC call: grpc.method=/v1alpha1.CSIDriverProvider/Mount
[signal SIGSEGV: segmentation violation code=0xaddress addr=0xaddress pc=0xaddress]

goroutine 26 [running]:
github.com/hashicorp/vault-csi-provider/internal/client.(*Client).auth(0xaddress, {0xaddress, 0xaddress}, 0xaddress, {0xaddress, 0xaddress})
	/home/runner/work/vault-csi-provider/vault-csi-provider/internal/client/client.go:159 +0xaddress
github.com/hashicorp/vault-csi-provider/internal/client.(*Client).RequestSecret(0xaddress, {0xaddress, 0xaddress}, 0xaddress, {{0xaddress, 0xaddress}, {0xaddress, 0xaddress}, {0xaddress, 0xaddress}, ...})
	/home/runner/work/vault-csi-provider/vault-csi-provider/internal/client/client.go:85 +0xaddress
github.com/hashicorp/vault-csi-provider/internal/provider.(*provider).getSecret(0xaddress, {0xaddress, 0xaddress}, 0xaddress, {{0xaddress, 0xaddress}, {0xaddress, 0xaddress}, {0xaddress, 0xaddress}, ...})
	/home/runner/work/vault-csi-provider/vault-csi-provider/internal/provider/provider.go:114 +0xaddress
github.com/hashicorp/vault-csi-provider/internal/provider.(*provider).HandleMountRequest(_, {_, _}, {{{0xaddress, 0xaddress}, {0xaddress, 0xaddress}, {0xaddress, 0xaddress}, {0xaddress, ...}, ...}, ...}, ...)
	/home/runner/work/vault-csi-provider/vault-csi-provider/internal/provider/provider.go:168 +0xaddress
github.com/hashicorp/vault-csi-provider/internal/server.(*Server).Mount(0xaddress, {0xaddress, 0xaddress}, 0xaddress?)
	/home/runner/work/vault-csi-provider/vault-csi-provider/internal/server/server.go:58 +0xaddress
sigs.k8s.io/secrets-store-csi-driver/provider/v1alpha1._CSIDriverProvider_Mount_Handler.func1({0xaddress?, 0xaddress?}, {0xaddress?, 0xaddress?})
	/home/runner/go/pkg/mod/sigs.k8s.io/secrets-store-csi-driver@v1.4.4/provider/v1alpha1/service_grpc.pb.go:139 +0xaddress
main.realMain.func1({0xaddress, 0xaddress}, {0xaddress, 0xaddress}, 0xaddress, 0xaddress)
	/home/runner/work/vault-csi-provider/vault-csi-provider/main.go:91 +0xaddress
sigs.k8s.io/secrets-store-csi-driver/provider/v1alpha1._CSIDriverProvider_Mount_Handler({0xaddress, 0xaddress}, {0xaddress, 0xaddress}, 0xaddress, 0xaddress)
	/home/runner/go/pkg/mod/sigs.k8s.io/secrets-store-csi-driver@v1.4.4/provider/v1alpha1/service_grpc.pb.go:141 +0xaddress
google.golang.org/grpc.(*Server).processUnaryRPC(0xaddress, {0xaddress, 0xaddress}, {0xaddress, 0xaddress}, 0xaddress, 0xaddress, 0xaddress, 0xaddress)
	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1379 +0xaddress
google.golang.org/grpc.(*Server).handleStream(0xaddress, {0xaddress, 0xaddress}, 0xaddress)
	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1790 +0xaddress
google.golang.org/grpc.(*Server).serveStreams.func2.1()
	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1029 +0xaddress
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 25
	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.65.0/server.go:1040 +0xaddress