Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Application insight profiler is not triggering for memory. No JFR report has been seen for memory. #3100

Open
Sachin1O1 opened this issue May 26, 2023 · 7 comments

Comments

@Sachin1O1
Copy link

Expected behavior

The profiler should trigger when the memory threshold is achieved and jfr report should be generated for memory.

Actual behavior

No JFR report is generated for memory.

To Reproduce

Submitting a PR with an example reproducing the issue in this repository would make it easier for the Application Insight maintainers to help you. Before doing this, you have to fork this repository.

System information

Please provide the following information:

  • SDK Version: 3.4.12
  • OS type and version: linux
  • Application Server type and version (if applicable):
  • Using spring-boot? yes
  • Additional relevant libraries (with version, if applicable):

Logs

Turn on SDK logs and attach/paste them to the issue. If using an application server, also attach any relevant server logs.

Be sure to remove any private information from the logs before posting!

Screenshots

If applicable, add screenshots to help explain your problem.
image

@trask
Copy link
Member

trask commented May 26, 2023

cc @johnoliver

@Sachin1O1
Copy link
Author

Any update?

cc @trask @johnoliver

@johnoliver
Copy link
Member

@Sachin1O1 its difficult to say without details, the best detail here would be a GC log, or a JFR file with the GC events included from a process that could allow us to confirm that your memory is over the threshold.

But some common issues could be:

  • You are triggering CPU profiles, this will also cause the cooldown to kick in and prevent re-triggering, this cooldown applies to ALL profiles, i.e if you have recently performed a CPU profile, a memory profile will refuse to trigger
  • As described at https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-profiler#memory
    • The percentage is how full the tenured region of your heap is, not the full heap
    • Occupancy is calculated as a percentage of the maximum possible tenured size not the current tenured size.
    • We only calculate the occupancy after a garbage collection involving the tenured region is executed (for instance, for G1 a mixed collection, or for the Parallel collector a full gc), i.e if your application is not performing tenured garbage collections it will not trigger.

If you can provide a gc log we can hopefully confirm that you are performing tenured garbage collections, and after the garbage collection is complete that your tenured occupancy is over the threshold.

@Sachin1O1
Copy link
Author

HI @johnoliver,
Thank you for explaining things. This gives a little clarity. If I understand it in the right way, you mean that there is no bug in the profiler triggering for memory and it should be something with the tenured garbage collections that I need to verify.

But I had an observation earlier that when the profiler trigger either for CPU or memory it shows the usage of other resources while it is triggered.

for example, if the profiler triggers for cpu at 90% in the report it also shows the value for memory usage at 40% but now it shows 0 as you can refer to the snap that I shared above in the issue. Please check this.

@johnoliver
Copy link
Member

@Sachin1O1

There might be a bug in the profiling trigger for memory, but I am not aware of one, and would need to see what the output of a tenured garbage collection looks like to confirm.

There is a bug we have where generally we only show the percentage of the resource that triggered the profile, and 0 for the other value. This is an issue that architecturally is difficult to work around, but is a known bug that we have. I will re-take a look at this to see if we can fix this

@Rohi-spark
Copy link

is there any updates. I am also facing same issue for container apps.

@johnoliver
Copy link
Member

johnoliver commented Jul 4, 2024

@Rohi-spark, are you able to confirm if:

  • Your tenured occupancy is breaching your threshold
    • Occupancy is calculated as the current occupancy of the tenured region vs the maximum possible size of the tenured region i.e if you requested a threshold of 40% and your tenured region has the possibility of growing to 1000mb, that your tenured region contains more than 400mb.
  • Your tenured region is above this threshold AFTER a tenured collection has happened. For G1 this will be mixed collections.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants