[VL] OOM on TPCDS q67 and q95

[jackylee-ch]

Describe the bug

java.lang.RuntimeException: Error during calling Java code from native code: java.lang.UnsupportedOperationException: Not enough spark off-heap execution memory. Acquired: 8388608, granted: 0. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application. 
	at io.glutenproject.memory.alloc.GlutenManagedReservationListener.reserveOrThrow(GlutenManagedReservationListener.java:53)
	at io.glutenproject.vectorized.ArrowOutIterator.nativeHasNext(Native Method)
	at io.glutenproject.vectorized.ArrowOutIterator.hasNextInternal(ArrowOutIterator.java:48)
	at io.glutenproject.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:37)
	at io.glutenproject.backendsapi.glutendata.GlutenIteratorApi$$anon$3.hasNext(GlutenIteratorApi.scala:300)
	at io.glutenproject.vectorized.CloseableColumnBatchIterator.hasNext(CloseableColumnBatchIterator.scala:41)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
	at org.apache.spark.shuffle.GlutenColumnarShuffleWriter.internalWrite(GlutenColumnarShuffleWriter.scala:96)
	at org.apache.spark.shuffle.GlutenColumnarShuffleWriter.write(GlutenColumnarShuffleWriter.scala:196)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:136)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:552)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1533)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:555)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

	at io.glutenproject.vectorized.ArrowOutIterator.nativeHasNext(Native Method)
	at io.glutenproject.vectorized.ArrowOutIterator.hasNextInternal(ArrowOutIterator.java:48)
	at io.glutenproject.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:37)
	at io.glutenproject.backendsapi.glutendata.GlutenIteratorApi$$anon$3.hasNext(GlutenIteratorApi.scala:300)
	at io.glutenproject.vectorized.CloseableColumnBatchIterator.hasNext(CloseableColumnBatchIterator.scala:41)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
	at org.apache.spark.shuffle.GlutenColumnarShuffleWriter.internalWrite(GlutenColumnarShuffleWriter.scala:96)
	at org.apache.spark.shuffle.GlutenColumnarShuffleWriter.write(GlutenColumnarShuffleWriter.scala:196)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:136)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:552)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1533)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:555)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

To Reproduce
spark version: 3.3.1
executor memory: 8g
executor cores: 8
executor memoryOverhead: 1g
offHeap.size: 24g

[rui-mo]

@jackylee-ch Thanks for reporting! Have you tried tweaking the offheap size to be larger? Looks the offheap memory is not enough.

[jackylee-ch]

I have changed the offheap from 24g to 32g, it still happened while running q67 and q95. We should fix current problem, otherwise blindly increasing the offheap size will lead to performance loss.

[rui-mo]

I have changed the offheap from 24g to 32g, it still happened while running q67 and q95. We should fix current problem, otherwise blindly increasing the offheap size will lead to performance loss.

@jackylee-ch Can I ask the dataset scale you are testing with? cc @zhztheplayer

[jackylee-ch]

oh, I tested with 10TB TPCDS, and we set useDoubleForDecimal and useStringForDate to true.

[rui-mo]

oh, I tested with 10TB TPCDS, and we set useDoubleForDecimal and useStringForDate to true.

@jackylee-ch Got it, that is large. We configure 46G offheap size for a 2T test, is that right? @zhouyuan

[flyfoxCI]

@zhouyuan can you share the test spark-submit parameters for tpcds?

[tianyouyangying]

Describe the bug

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 436 in stage 25.0 failed 4 times, most recent failure: Lost task 436.3 in stage 25.0 (TID 15488) (node72 executor 16): java.lang.RuntimeException: Error during calling Java code from native code: io.glutenproject.memory.alloc.ManagedReservationListener$OutOfMemoryException: Not enough spark off-heap execution memory. Acquired: 16777216, granted: 11184810. Try tweaking config option spark.memory.offHeap.size to get larger space to run this application. 
		at io.glutenproject.memory.alloc.ManagedReservationListener.lambda$reserveOrThrow$0(ManagedReservationListener.java:78)
		at io.glutenproject.memory.alloc.ManagedReservationListener.reserve0(ManagedReservationListener.java:55)
		at io.glutenproject.memory.alloc.ManagedReservationListener.reserveOrThrow(ManagedReservationListener.java:75)
		at io.glutenproject.vectorized.ShuffleWriterJniWrapper.split(Native Method)
		at org.apache.spark.shuffle.ColumnarShuffleWriter.internalWrite(ColumnarShuffleWriter.scala:166)
		at org.apache.spark.shuffle.ColumnarShuffleWriter.write(ColumnarShuffleWriter.scala:217)
		at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
		at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
		at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
		at org.apache.spark.scheduler.Task.run(Task.scala:136)
		at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
		at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
		at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
		at java.lang.Thread.run(Thread.java:748)
	
		at io.glutenproject.vectorized.ShuffleWriterJniWrapper.split(Native Method)
		at org.apache.spark.shuffle.ColumnarShuffleWriter.internalWrite(ColumnarShuffleWriter.scala:166)
		at org.apache.spark.shuffle.ColumnarShuffleWriter.write(ColumnarShuffleWriter.scala:217)
		at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
		at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
		at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
		at org.apache.spark.scheduler.Task.run(Task.scala:136)
		at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
		at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
		at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
		at java.lang.Thread.run(Thread.java:748)

To Reproduce
spark version: 3.3.1
executor memory: 10g
executor cores: 4
executor memoryOverhead: 2g
offHeap.size: 20g

[zhztheplayer]

Would you like to share Gluten's commit hash you are using? Thanks.