-
Notifications
You must be signed in to change notification settings - Fork 859
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Modifying ExecutorGC heuristic help page to add more detailed suggest…
…ions
- Loading branch information
1 parent
24c3fd3
commit 927c7c8
Showing
1 changed file
with
9 additions
and
4 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -17,7 +17,12 @@ | |
<p>This analysis shows how much time a job is spending in GC. To normalise the results across all jobs, the ratio of the time a job spends in Gc to the total run time of the job is calculated. </p> | ||
<p>A job is flagged if the ratio is too high, meaning the job spends too much time in GC.</p> | ||
<h3>Suggestions</h3> | ||
<p>We recommend increasing the executor memory.</p> | ||
<p>Enabling ParallelGC using spark.executor.extraJavaOptions could help.</p> | ||
<p>Also recommended to reduce the number of UDF calls.</p> | ||
<p>For more help refer <a href="https://iwww.corp.linkedin.com/wiki/cf/display/DWH/Spark+SQL+Tuning+Techniques" target="_blank">here</a></p> | ||
<ul> | ||
<li>We recommend increasing the executor memory.</li> | ||
<li>Enabling ParallelGC using spark.executor.extraJavaOptions could help.</li> | ||
This comment has been minimized.
Sorry, something went wrong.
This comment has been minimized.
Sorry, something went wrong.
ShubhamGupta29
Author
Contributor
|
||
<ul> | ||
<li>User can enable G1GC or ParallelGC by adding <b>-XX:+UseG1GC</b> or <b>-XX:+UseParallelGC</b> respectively to Spark configuration spark.executor.extraJavaOptions</li> | ||
</ul> | ||
<li>High GC can occur if the number of UDF calls made is high, especially if the UDFs are inefficient or use a lot of memory.</li> | ||
</ul> | ||
<p>For some general guideline about how to tune GC for your Spark application refer <a href="https://spark.apache.org/docs/latest/tuning.html#garbage-collection-tuning" target="_blank">here</a></p> |
Please change to "Enabling ParallelGC or G1GC".