PerfView Unsure How To Figure Out Performance Issue

[justalexh]

Hello all,

I'm testing a microservice system on Azure:

• vP2v3 is the service plan
• API testing

Issue

• I'm unable to figure out what the actual cause is when it comes to my performance issue, I've tried looking at the threads that are blocked.
• I decided to dig deep into seeing which functions are being called after SwapContext happens and when the BLOCKED_TIME occurs.
• I have attempted to look between the range for when JIT calls happened as well, but don't have any idea if it really is the cause of performance issues.
  • Most of the Jit calls are under 0.5ms, but there is a few occasional ones that popup anywhere from 5,000 - 40,000 msec

Information: Thread 11296

• In this case, I often see ETW methods being called and I'm not sure if that's due to having multiple monitoring systems logging/profiling the system.
• I often seeMCGenControlCallBackV2 but can't find much information about it despite it being apparent in the path.

Information: Thread 8204

• This one has a LOT of path's so please bear with me.

System.Threading.SpinWait.SpinOnceCore and System.Threading.Tasks.Task.SpinWait(int32)

CancellationTokenSource.InitializerWithTimer(unsigned int32)

• We added a cancellation token due to having time outs after 1 minute, but the requests are still going in on the background after it times out.

CoreclrThePreStub and System.Net.Http.HttpConnection.Bool

HttpConnectionPool;;+<ConnectAsync>.MoveNext()

`coreclr!OnCallCountThresholdReachedstub2

Conclusion

I apologize for all the images, but I'm fairly new to PerfView and Performance so any guidance would help greatly.

[brianrob]

@justalexh, can you share a quick description of the setup of your app as well as a description of the performance issue that you're seeing. Looking at the images is difficult because I can't quite get a full picture - it would be best if you're able to share the trace. If you would prefer, you can also e-mail it to me at brianrob [at] microsoft.

From the images, the one thing that caught my attention is the time spent in OnCallCountThresholdReached - there appears to be a hot lock there, which is unusual. This would cause delays for methods as they are tiered up (tiered compilation).

[justalexh]

@brianrob , thank you for the response. I have emailed you

[justalexh]

@brianrob Also, could you explain to me what a hot lock is and why it's unusual and how you noticed it?

What exactly is tiered compilation in this case?

[brianrob]

@justalexh, it looks like I did not receive an e-mail from you. Can you please re-send to brianrob [at] microsoft [dot] com? Thanks.