You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I want to know why we need to do rdd collect here. Doing this kind of operation in spark will put a lot of pressure on the driver. During the test, I found that the driver often reports OOM.
When I check the source code, the spark driver performs the commit operation of the coordinator_olnly type stage. I understand that only metadata needs to be submitted when doing the commit operation. Why do I need to submit the data together.
Is the coordinator implemented in this way in native presto?
This discussion was converted from issue #23830 on October 15, 2024 13:54.
Heading
Bold
Italic
Quote
Code
Link
Numbered list
Unordered list
Task list
Attach files
Mention
Reference
Menu
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
I want to know why we need to do rdd collect here. Doing this kind of operation in spark will put a lot of pressure on the driver. During the test, I found that the driver often reports OOM.
When I check the source code, the spark driver performs the commit operation of the coordinator_olnly type stage. I understand that only metadata needs to be submitted when doing the commit operation. Why do I need to submit the data together.
Is the coordinator implemented in this way in native presto?
Beta Was this translation helpful? Give feedback.
All reactions