[GR-52400] Revise heuristic for memory usage of Native Image build process. #10441
Conversation
oracle-contributor-agreement
bot
added
the
OCA Verified
Follow up to #7440
Using `svm.` as a prefix makes it clearer this is internal API.
graalvmbot
force-pushed
the
fniephaus/GR-52400/memory-usage
branch
2 times, most recently
from
019ab36 to
ee2c4d1
Compare
graalvmbot
changed the title
graalvmbot
force-pushed
the
fniephaus/GR-52400/memory-usage
branch
from
ee2c4d1 to
9ef987f
Compare
It shows KB/MB/GB but calculated KiB/MiB/GiB.
graalvmbot
force-pushed
the
fniephaus/GR-52400/memory-usage
branch
from
9ef987f to
e424fb5
Compare
substratevm/src/com.oracle.svm.driver/src/com/oracle/svm/driver/MemoryUtil.java
Outdated
Show resolved
Hide resolved
| memoryUsageReason = "enough available"; | ||
| } else { // fall back to dedicated mode | ||
| memoryUsageReason = "not enough available"; |
There was a problem hiding this comment.
"(not) enough available" seems confusing to me.
- "enough available" makes me think I should not tune anything even if my build fails due to out of memory or I get a lot of time spent on GC. I would replace this with something like "available free memory used".
- "not enough available" makes me feel something is wrong and I need to tune things. I would replace this with something like "less than 8GB ram is available, setting mac to 85% of system memory".
There was a problem hiding this comment.
+1. Mentioning the threshold value of 8G would be helpful to the user.
There was a problem hiding this comment.
Yes, we can do that. I also tried to keep the text short so that the lines fit in 80 characters. I'll see if I can come up with something better.
There was a problem hiding this comment.
Changed it to:
- 13.24GB of memory (85.0% of system memory, less than 8GB of memory available)
for now.
| memoryUsageReason = "enough available"; | ||
| } else { // fall back to dedicated mode | ||
| memoryUsageReason = "not enough available"; |
There was a problem hiding this comment.
+1. Mentioning the threshold value of 8G would be helpful to the user.
| */ | ||
| reasonableMaxMemorySize = totalMemorySize * DEDICATED_MODE_TOTAL_MEMORY_RATIO; | ||
| String memoryUsageReason = "unknown"; | ||
| final boolean isDedicatedMemoryUsage; |
There was a problem hiding this comment.
It looks like there is no way for a developer to chose shared mode explicitly (only happens if $CI != true and not containerized). dedicated can be forced with CI=true, but not so for shared. Maybe that would be worth doing?
There was a problem hiding this comment.
You are essentially asking for #8468. I thought about this some more and it didn't seem worth the additional complexity (needs special option treatment because it happens very early in the process). Why would anyone force available memory when that is a moving target? Also, you can easily override the default behavior with Xmx and other memory flags.
There was a problem hiding this comment.
Sure. No strong feelings about this.
This PR mainly revises the heuristic that is used, unless overridden by flags, to determine the memory limits of the Native Image build process. The heuristic selects one for two modes:
Unless overridden with Xmx or other memory flags, the heuristic will use dedicated mode in containers and CI environments. Otherwise, it will try to use available memory except if there is not enough memory available (in that case it falls back to the dedicated mode).
The PR also adjusts the build resource section of the build output so that it shows a reason for selected memory limit.
In addition, the PR also addresses a number of smaller issues (see individual commits).