Skip to content

Commit

Permalink
Modifying ExecutorGC heuristic help page to add more detailed suggest…
Browse files Browse the repository at this point in the history
…ions
  • Loading branch information
ShubhamGupta29 committed Oct 12, 2018
1 parent 24c3fd3 commit 927c7c8
Showing 1 changed file with 9 additions and 4 deletions.
13 changes: 9 additions & 4 deletions app/views/help/spark/helpExecutorGcHeuristic.scala.html
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,12 @@
<p>This analysis shows how much time a job is spending in GC. To normalise the results across all jobs, the ratio of the time a job spends in Gc to the total run time of the job is calculated. </p>
<p>A job is flagged if the ratio is too high, meaning the job spends too much time in GC.</p>
<h3>Suggestions</h3>
<p>We recommend increasing the executor memory.</p>
<p>Enabling ParallelGC using spark.executor.extraJavaOptions could help.</p>
<p>Also recommended to reduce the number of UDF calls.</p>
<p>For more help refer <a href="https://iwww.corp.linkedin.com/wiki/cf/display/DWH/Spark+SQL+Tuning+Techniques" target="_blank">here</a></p>
<ul>
<li>We recommend increasing the executor memory.</li>
<li>Enabling ParallelGC using spark.executor.extraJavaOptions could help.</li>

This comment has been minimized.

Copy link
@edwinalu

edwinalu Oct 12, 2018

Please change to "Enabling ParallelGC or G1GC".

This comment has been minimized.

Copy link
@ShubhamGupta29

ShubhamGupta29 Oct 14, 2018

Author Contributor

Done and also added a check in the heuristic class for G1GC for users.

<ul>
<li>User can enable G1GC or ParallelGC by adding <b>-XX:+UseG1GC</b> or <b>-XX:+UseParallelGC</b> respectively to Spark configuration spark.executor.extraJavaOptions</li>
</ul>
<li>High GC can occur if the number of UDF calls made is high, especially if the UDFs are inefficient or use a lot of memory.</li>
</ul>
<p>For some general guideline about how to tune GC for your Spark application refer <a href="https://spark.apache.org/docs/latest/tuning.html#garbage-collection-tuning" target="_blank">here</a></p>

0 comments on commit 927c7c8

Please sign in to comment.