[Core] Support Jvm memory shrinking for DynamicOffHeapSizingMemoryTarget #9585

zhli1142015 · 2025-05-09T15:07:08Z

What changes were proposed in this pull request?

Support Jvm memory shrinking for DynamicOffHeapSizingMemoryTarget
When a memory request cannot be satisfied, we first lower the value of option MaxHeapFreeRatio and trigger a full GC to force the JVM to return as much memory as possible back to the operating system. This approach incurs the overhead of a full GC each time.
Currently, only Java 11 and Java 17 are supported. And this has only been validated with the G1 GC.
Config

......
"spark.memory.offHeap.enabled": "false",
"spark.gluten.memory.dynamic.offHeap.sizing.memory.fraction": "0.95",
"spark.gluten.memory.dynamic.offHeap.sizing.enabled" "true"
......

JVM memory shrinking

2025-05-09 09:35:46,862 WARN DynamicOffHeapSizingMemoryTarget [gc-thread-pool]: Starting async full gc to shrink JVM memory: Total On-heap: 58074333184, Free On-heap: 33397024368, Total Off-heap: 8388608, Used On-Heap: 24677308816, Executor memory: 60129542144.
2025-05-09 09:35:48,811 WARN DynamicOffHeapSizingMemoryTarget [gc-thread-pool]: Finished async full gc to shrink JVM memory: Total On-heap: 1199570944, Free On-heap: 116309888, Total Off-heap: 8388608, Used On-Heap: 1083261056, Executor memory: 60129542144, [GC Retry times: 0].

github-actions · 2025-05-09T15:07:29Z

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

Other pull requests

github-actions · 2025-05-09T15:07:39Z

Run Gluten Clickhouse CI on x86

zhli1142015 · 2025-05-09T15:08:14Z

cc @FelixYBW and @zhztheplayer , thanks.

github-actions · 2025-05-09T15:12:27Z

Run Gluten Clickhouse CI on x86

zhztheplayer · 2025-05-12T12:59:55Z

Thank you for iterating this feature!

Curious, are there any future plans on the feature from your end? Or this is the final revision?

zhli1142015 · 2025-05-12T13:11:30Z

From a functionality standpoint, this covers everything for now. We may add some monitoring and debugging improvements later, and adjust the strategies used based on customer feedback.

github-actions · 2025-05-12T14:51:09Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-05-12T14:54:43Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-05-13T02:28:29Z

Run Gluten Clickhouse CI on x86

zhli1142015 · 2025-05-13T02:35:13Z

Is there some thing wrong with the build pipeline? cc @zhouyuan and @FelixYBW
https://github.com/apache/incubator-gluten/actions/runs/14986805726/job/42102259698?pr=9585

/usr/bin/docker pull apache/gluten:centos-8-jdk8
  centos-8-jdk8: Pulling from apache/gluten
  no matching manifest for linux/amd64 in the manifest list entries
  Warning: Docker pull failed with exit code 1, back off 4.4[17](https://github.com/apache/incubator-gluten/actions/runs/14986809352/job/42102270035?pr=9596#step:2:20) seconds before retry.
  /usr/bin/docker pull apache/gluten:centos-8-jdk8
  centos-8-jdk8: Pulling from apache/gluten
  no matching manifest for linux/amd64 in the manifest list entries
  Warning: Docker pull failed with exit code 1, back off 1.601 seconds before retry.
  /usr/bin/docker pull apache/gluten:centos-8-jdk8
  centos-8-jdk8: Pulling from apache/gluten
  no matching manifest for linux/amd64 in the manifest list entries
  Error: Docker pull failed with exit code 1

zhztheplayer

Haven't got time to take a closer look on this but feel free to merge if you are confident.

By the way, it feels we'd have basic spilling support for the feature to make it usable? Any thoughts on that?

zhli1142015 · 2025-05-15T04:59:21Z

By the way, it feels we'd have basic spilling support for the feature to make it usable? Any thoughts on that?

Thanks for pointing this out. Through testing I found that with this PR the spill logic is still functioning. In our tests we observed that once the dynamic memory manager frees enough JVM memory, if the native task still needs more memory, spilling will be triggered as expected.
Please let me know if I have missed anything here.

github-actions · 2025-05-26T07:39:15Z

Run Gluten Clickhouse CI on x86

zhztheplayer · 2025-05-26T11:11:14Z

Thanks for pointing this out. Through testing I found that with this PR the spill logic is still functioning. In our tests we observed that once the dynamic memory manager frees enough JVM memory, if the native task still needs more memory, spilling will be triggered as expected.
Please let me know if I have missed anything here.

Thanks for the inputs. I thought the feature would not trigger spill correctly since it was not reporting the usage to Spark (Spark triggers Velox spilling), maybe I missed something there. Will have a look once available. (non-blocking)

github-actions · 2025-05-26T11:41:30Z

Run Gluten Clickhouse CI on x86

zhli1142015 · 2025-05-27T03:12:27Z

This PR previously missed a change: when this feature is enabled, the memory allocations that Gluten reports to Spark should be counted against the ON_HEAP quota rather than OFF_HEAP. From my understanding, this change is the key for this feature to report memory usage correctly; with it, spilling will work as expected.

github-actions · 2025-05-28T05:08:39Z

Run Gluten Clickhouse CI on x86

zhouyuan

Thanks! I just learned more details on this feature

...-core/src/main/java/org/apache/gluten/memory/memtarget/DynamicOffHeapSizingMemoryTarget.java

backends-velox/src/test/scala/org/apache/gluten/execution/DynamicOffHeapSizingSuite.scala

...-core/src/main/java/org/apache/gluten/memory/memtarget/DynamicOffHeapSizingMemoryTarget.java

github-actions · 2025-05-28T11:40:56Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-05-28T12:05:51Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-05-29T02:50:42Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-05-30T03:53:37Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-06-05T02:36:26Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-06-10T02:36:23Z

Run Gluten Clickhouse CI on x86

github-actions · 2025-07-09T04:37:13Z

Run Gluten Clickhouse CI on x86

github-actions bot added CORE works for Gluten Core VELOX labels May 9, 2025

zhli1142015 requested review from zhztheplayer and zhouyuan May 12, 2025 12:47

zhli1142015 force-pushed the jvm_shrink branch from ae36606 to f8f48dc Compare May 12, 2025 14:50

zhli1142015 force-pushed the jvm_shrink branch from f8f48dc to a425f38 Compare May 12, 2025 14:54

zhztheplayer approved these changes May 14, 2025

View reviewed changes

zhli1142015 force-pushed the jvm_shrink branch from c6647ec to baeea78 Compare May 26, 2025 07:38

zhli1142015 force-pushed the jvm_shrink branch from 9282b4e to 96b9810 Compare May 28, 2025 05:08

zhouyuan approved these changes May 28, 2025

View reviewed changes

zhli1142015 force-pushed the jvm_shrink branch from 4627438 to 96b9810 Compare May 28, 2025 12:05

zhli1142015 force-pushed the jvm_shrink branch from 96b9810 to c2fcb04 Compare May 29, 2025 02:50

zhli1142015 force-pushed the jvm_shrink branch from c2fcb04 to fc11e64 Compare May 30, 2025 03:53

zhli1142015 force-pushed the jvm_shrink branch from fc11e64 to 50592a4 Compare June 5, 2025 02:35

zhli1142015 force-pushed the jvm_shrink branch from 50592a4 to c4e727a Compare June 10, 2025 02:35

zhli1142015 added 5 commits July 9, 2025 12:19

[Core] Support Jvm memory shrinking for DynamicOffHeapSizingMemoryTarget

106c853

minor change

0b942f7

minor change

50daaa1

fix ut

90a2fcf

fix build

c61aa42

zhli1142015 force-pushed the jvm_shrink branch from c4e727a to c61aa42 Compare July 9, 2025 04:36

zhli1142015 merged commit 07b18e7 into apache:main Jul 9, 2025
50 checks passed

[Core] Support Jvm memory shrinking for DynamicOffHeapSizingMemoryTarget #9585

[Core] Support Jvm memory shrinking for DynamicOffHeapSizingMemoryTarget #9585

Uh oh!

Conversation

zhli1142015 commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Uh oh!

github-actions bot commented May 9, 2025

Uh oh!

github-actions bot commented May 9, 2025

Uh oh!

zhli1142015 commented May 9, 2025

Uh oh!

github-actions bot commented May 9, 2025

Uh oh!

zhztheplayer commented May 12, 2025

Uh oh!

zhli1142015 commented May 12, 2025

Uh oh!

github-actions bot commented May 12, 2025

Uh oh!

github-actions bot commented May 12, 2025

Uh oh!

github-actions bot commented May 13, 2025

Uh oh!

zhli1142015 commented May 13, 2025

Uh oh!

zhztheplayer left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhli1142015 commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented May 26, 2025

Uh oh!

zhztheplayer commented May 26, 2025

Uh oh!

github-actions bot commented May 26, 2025

Uh oh!

zhli1142015 commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented May 28, 2025

Uh oh!

zhouyuan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented May 28, 2025

Uh oh!

github-actions bot commented May 28, 2025

Uh oh!

github-actions bot commented May 29, 2025

Uh oh!

github-actions bot commented May 30, 2025

Uh oh!

github-actions bot commented Jun 5, 2025

Uh oh!

github-actions bot commented Jun 10, 2025

Uh oh!

github-actions bot commented Jul 9, 2025

Uh oh!

Uh oh!

Uh oh!

zhli1142015 commented May 9, 2025 •

edited

Loading

zhztheplayer left a comment •

edited

Loading

zhli1142015 commented May 15, 2025 •

edited

Loading

zhli1142015 commented May 27, 2025 •

edited

Loading