Lot of ResourceRequestsThrottled

[ahurtaud]

Hello,

On some big subscriptions we are seeing in promitor scrapers a lot of:

[15:57:55 FTL] ResourceRequestsThrottled: Number of requests for action 'microsoft.insights/metricDefinitions/read' exceeded the limit of '400' for time interval '00:05:00'. Please try again after '309' seconds.
[15:57:55 FTL] ResourceRequestsThrottled: Number of requests for action 'microsoft.insights/metricDefinitions/read' exceeded the limit of '400' for time interval '00:05:00'. Please try again after '309' seconds.
[15:57:55 FTL] ResourceRequestsThrottled: Number of requests for action 'microsoft.insights/metricDefinitions/read' exceeded the limit of '400' for time interval '00:05:00'. Please try again after '309' seconds.

What this microsoft.insights/metricDefinitions/read is used for? and how could we reduce it if possible?

Thank you

[tomkerkhove]

How many resources are you scraping? This could be Azure throttling not being handled well.

[ahurtaud]

I think the main points are for 560 network interfaces and 141 storage accounts.

[tomkerkhove]

That might be it, can you check the throttling metrics on the agent to see if they give an indication in this area?

[ahurtaud]

Scraper:
Lots of throttling logs (the one from first post about 400 calls every 5mins), nothing about the 12k read/hour.

Well the promitor_ratelimit_arm (first graph seems ok for this subscription) is saying that I have a lot of remaining calls.
But promitor_ratelimit_arm_throttled (second graph) says I am most of the time throttled=true

Resource_discovery:
Third graph is resource discovery / azure graph. All good. no throttling log.

[tomkerkhove]

Odd, we might be doing too many calls at once. Feel free to open an issue on this, but it will be a hard one to reproduce I think.

[ahurtaud]

#1891