[SPARK-34965][BUILD] Remove .sbtopts that duplicately sets the default memory by HyukjinKwon · Pull Request #32062 · apache/spark

HyukjinKwon · 2021-04-06T11:17:50Z

What changes were proposed in this pull request?

This PR removes .sbtopts (added in #29286) that duplicately sets the default memory. The default memories are set:

Lines 119 to 124 in 3b634f6

    
           local mem=${1:-$sbt_default_mem} 
        
           local codecache=$(( $mem / 8 )) 
        
           (( $codecache > 128 )) || codecache=128 
        
           (( $codecache < 2048 )) || codecache=2048 
        
           echo "-Xms${mem}m -Xmx${mem}m -XX:ReservedCodeCacheSize=${codecache}m"

Why are the changes needed?

This file disables the memory option from the build/sbt script:

./build/sbt -mem 6144

.../jdk-11.0.3.jdk/Contents/Home as default JAVA_HOME.
Note, this will be overridden by -java-home if it is set.
Error occurred during initialization of VM
Initial heap size set to a larger value than the maximum heap size

because it adds these memory options at the last:

/.../bin/java -Xms6144m -Xmx6144m -XX:ReservedCodeCacheSize=256m -Xmx4G -Xss4m -jar build/sbt-launch-1.5.0.jar

and Java respects the rightmost memory configurations.

Does this PR introduce any user-facing change?

No, dev-only.

How was this patch tested?

Manually ran SBT. It will be tested in the CIs in this Pr.

SparkQA · 2021-04-06T12:05:38Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41517/

cloud-fan

makes sense

SparkQA · 2021-04-06T12:54:33Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41519/

HyukjinKwon · 2021-04-06T13:20:53Z

build/sbt-launch-lib.bash

 declare -a sbt_commands
 declare -a maven_profiles
-declare sbt_default_mem=2048
+declare sbt_default_mem=4096


So here was the default place to set the memory

SparkQA · 2021-04-06T13:52:31Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41522/

SparkQA · 2021-04-06T14:05:48Z

Test build #136940 has finished for PR 32062 at commit dafae34.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-06T14:18:49Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41523/

SparkQA · 2021-04-06T14:18:50Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41523/

SparkQA · 2021-04-06T14:42:17Z

Test build #136943 has finished for PR 32062 at commit 2523b55.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-06T15:44:14Z

Test build #136946 has finished for PR 32062 at commit 7a6959a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-06T15:49:18Z

Test build #136947 has finished for PR 32062 at commit 66987d1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…t memory ### What changes were proposed in this pull request? This PR removes `.sbtopts` (added in #29286) that duplicately sets the default memory. The default memories are set: https://github.com/apache/spark/blob/3b634f66c3e4a942178a1e322ae65ce82779625d/build/sbt-launch-lib.bash#L119-L124 ### Why are the changes needed? This file disables the memory option from the `build/sbt` script: ```bash ./build/sbt -mem 6144 ``` ``` .../jdk-11.0.3.jdk/Contents/Home as default JAVA_HOME. Note, this will be overridden by -java-home if it is set. Error occurred during initialization of VM Initial heap size set to a larger value than the maximum heap size ``` because it adds these memory options at the last: ```bash /.../bin/java -Xms6144m -Xmx6144m -XX:ReservedCodeCacheSize=256m -Xmx4G -Xss4m -jar build/sbt-launch-1.5.0.jar ``` and Java respects the rightmost memory configurations. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Manually ran SBT. It will be tested in the CIs in this Pr. Closes #32062 from HyukjinKwon/SPARK-34965. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 2eda1c6) Signed-off-by: Dongjoon Hyun <[email protected]>

dongjoon-hyun · 2021-04-06T22:18:04Z

Thank you, @HyukjinKwon and all.
Merged to master/3.1.

HyukjinKwon · 2021-04-07T00:05:59Z

Thanks!

…t memory ### What changes were proposed in this pull request? This PR removes `.sbtopts` (added in apache#29286) that duplicately sets the default memory. The default memories are set: https://github.com/apache/spark/blob/3b634f66c3e4a942178a1e322ae65ce82779625d/build/sbt-launch-lib.bash#L119-L124 ### Why are the changes needed? This file disables the memory option from the `build/sbt` script: ```bash ./build/sbt -mem 6144 ``` ``` .../jdk-11.0.3.jdk/Contents/Home as default JAVA_HOME. Note, this will be overridden by -java-home if it is set. Error occurred during initialization of VM Initial heap size set to a larger value than the maximum heap size ``` because it adds these memory options at the last: ```bash /.../bin/java -Xms6144m -Xmx6144m -XX:ReservedCodeCacheSize=256m -Xmx4G -Xss4m -jar build/sbt-launch-1.5.0.jar ``` and Java respects the rightmost memory configurations. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Manually ran SBT. It will be tested in the CIs in this Pr. Closes apache#32062 from HyukjinKwon/SPARK-34965. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit 2eda1c6) Signed-off-by: Dongjoon Hyun <[email protected]>

HyukjinKwon changed the title ~~[kSPARK-34965][BUILD] Remove .sbtopts that duplicately sets the default memory~~ [SPARK-34965][BUILD] Remove .sbtopts that duplicately sets the default memory Apr 6, 2021

github-actions bot added the DOCS label Apr 6, 2021

cloud-fan approved these changes Apr 6, 2021

View reviewed changes

srowen approved these changes Apr 6, 2021

View reviewed changes

Remove .sbtopts that duplicately sets the default memory

7a6959a

HyukjinKwon force-pushed the SPARK-34965 branch from 2523b55 to 7a6959a Compare April 6, 2021 12:39

github-actions bot added the BUILD label Apr 6, 2021

Match the memory setting

66987d1

HyukjinKwon commented Apr 6, 2021

View reviewed changes

dongjoon-hyun approved these changes Apr 6, 2021

View reviewed changes

dongjoon-hyun closed this in 2eda1c6 Apr 6, 2021

HyukjinKwon deleted the SPARK-34965 branch January 4, 2022 00:54

	local mem=${1:-$sbt_default_mem}
	local codecache=$(( $mem / 8 ))
	(( $codecache > 128 )) \|\| codecache=128
	(( $codecache < 2048 )) \|\| codecache=2048

	echo "-Xms${mem}m -Xmx${mem}m -XX:ReservedCodeCacheSize=${codecache}m"

Conversation

HyukjinKwon commented Apr 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Apr 6, 2021

Uh oh!

cloud-fan left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 6, 2021

Uh oh!

HyukjinKwon Apr 6, 2021

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 6, 2021

Uh oh!

SparkQA commented Apr 6, 2021

Uh oh!

SparkQA commented Apr 6, 2021

Uh oh!

SparkQA commented Apr 6, 2021

Uh oh!

SparkQA commented Apr 6, 2021

Uh oh!

SparkQA commented Apr 6, 2021

Uh oh!

SparkQA commented Apr 6, 2021

Uh oh!

dongjoon-hyun commented Apr 6, 2021

Uh oh!

HyukjinKwon commented Apr 7, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

HyukjinKwon commented Apr 6, 2021 •

edited

Loading