Informational messages are logged as errors

[fubar]

Our logs show the below messages logged frequently and at error level:

I0322 06:00:50.813426       1 server.go:107] Servicing mount request for pod [...] in namespace default using service account [...] with region eu-west-1
I0322 06:00:51.010475       1 auth.go:123] Role ARN for [...] is arn:aws:iam::[...]

These are informational messages that should not be treated like errors.

[pierluigilenoci]

@fubar these are informational messages.
Why do you say they are treated like errors?

Keep in mind that Klog (the library used for logging) is affected by a fairly serious bug regarding error levels.
And I don't see any fixes on the horizon anytime soon.

Ref: kubernetes/klog#212

[fubar]

Interesting, that may as well be the cause. We're using a standard Datadog agent setup for logging, and all of those log events are logged at the error level in Datadog.

Does the AWS provider have any configuration options for logging?

[pierluigilenoci]

@fubar The CSI driver yes, the AWS provider no.

[fubar]

I'll try out the logFormatJSON option tomorrow, thanks!

[fubar]

I tried out the available options of the CSI driver, but they have no effect on the log level. Everything still gets logged as errors. Unfortunately, the only option we have is to not log anything the driver and provider are outputting.

[pierluigilenoci]

@fubar I quote myself.

Keep in mind that Klog (the library used for logging) is affected by a fairly serious bug regarding error levels. And I don't see any fixes on the horizon anytime soon.

Ref: kubernetes/klog#212

[simonmarty]

We're waiting on the kubernetes team to fix the klog issue mentioned. Feel free to reopen when the upstream issue has been addressed.