failed to waitForNotification: callbackNumber does not exist in callbackMap

[jkroepke]

Hi,

xref:

• Troubleshooting Windows Exporter on EKS: Metrics Missing for Exporter Container and Callback/Connection Errors in Logs prometheus-community/windows_exporter#1861

a EKS user recently reports a issue with the windows_exporter.

[...]
windows-exporter time="2025-01-29T14:07:57Z" level=error msg="failed to waitForNotification: callbackNumber does not exist in callbackMap" callbackNumber=1095
windows-exporter time="2025-01-29T14:07:57Z" level=error msg="failed to waitForNotification: callbackNumber does not exist in callbackMap" callbackNumber=1096
windows-exporter time="2025-01-29T14:07:57Z" level=error msg="failed to waitForNotification: callbackNumber does not exist in callbackMap" callbackNumber=1097
windows-exporter time="2025-01-29T14:07:57Z" level=error msg="failed to waitForNotification: callbackNumber does not exist in callbackMap" callbackNumber=1098
windows-exporter time="2025-01-29T14:07:57Z" level=error msg="failed to waitForNotification: callbackNumber does not exist in callbackMap" callbackNumber=1099
windows-exporter time="2025-01-29T14:08:12Z" level=error msg="failed to waitForNotification: callbackNumber does not exist in callbackMap" callbackNumber=1100
windows-exporter time="2025-01-29T14:08:12Z" level=error msg="failed to waitForNotification: callbackNumber does not exist in callbackMap" callbackNumber=1102
windows-exporter time="2025-01-29T14:08:12Z" level=error msg="failed to waitForNotification: callbackNumber does not exist in callbackMap" callbackNumber=1103
windows-exporter time="2025-01-29T14:08:12Z" level=error msg="failed to waitForNotification: callbackNumber does not exist in callbackMap" callbackNumber=1104
windows-exporter time="2025-01-29T14:08:12Z" level=error msg="failed to waitForNotification: callbackNumber does not exist in callbackMap" callbackNumber=1105
windows-exporter time="2025-01-29T14:08:12Z" level=error msg="failed to waitForNotification: callbackNumber does not exist in callbackMap" callbackNumber=1106
windows-exporter time="2025-01-29T14:08:12Z" level=error msg="failed to waitForNotification: callbackNumber does not exist in callbackMap" callbackNumber=1109
windows-exporter time="2025-01-29T14:08:12Z" level=error msg="failed to waitForNotification: callbackNumber does not exist in callbackMap" callbackNumber=1108
[...]

After a quick research, the messages are coming from

hcsshim/internal/hcs/waithelper.go

Line 37 in 8d81359

log.G(ctx).WithField("callbackNumber", callbackNumber).Error("failed to waitForNotification: callbackNumber does not exist in callbackMap")

As maintainer of windows_exporter, I would like to ask how I can identify the root cause? Or do I implement hcsshim wrongly?

• Windows: Server 2019
• hcsshim version: github.com/Microsoft/hcsshim v0.12.7

It is noticeable, that the same error appears with v0.12.5 as well.