Build: Allow overriding the default test parallelism of 1 #13675
+14
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds the `testParallelism` property which allows to set the number of Gradle workers for test tasks. The default remains 1 worker (no concurrency). I found myself frequently editing the `test` section of the Flink builds to speed up the build. The speedup is linear to the number of workers used, given that the machine can keep up. Example: ``` ./gradlew :iceberg-flink:iceberg-flink-2.0:check -PtestParallelism=8 ``` Initially, I was planning to add this only to the Flink build files, but I think all other modules of Iceberg can benefit from this build setting.
|
I had been manually editing this as well. |
mxm
force-pushed
the
test-parallelism
branch
2 times, most recently
from
a45fbfb to
f2f3c33
Compare
This reverts commit 12e1a59.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds the
testParallelismproperty which allows to set the number of Gradle workers for test tasks. The default remains 1 worker (no concurrency).I found myself frequently editing the
testsection of the Flink build files to speed up the build. This PR allows everyone in Iceberg to selectively increase the test parallelism.Why?
While the build tasks run in parallel, there is no concurrency on the individual build tasks. By far, tests take the longest time to execute. Running them in parallel yields a speedup of at least 2. Locally, speedups of 3 are possible. In CI, the speedup is also above 2. See the test results below.
How?
Set the number of test workers to a fixed number:
Enable auto-mode, which sets the test parallelism to
availableCPUs / 2:Test results
Local
On my machine (MacBook Pro M1 Max):
Note:
GitHub CI
In this PR, I enabled the flag to run Flink tests. I don't know how many cores we have available in CI, so I tested only with the status quo (
testParallelism=1) and the auto-mode set tonumCpusandnumCpus/2(the latter is what I propose as the default for the auto mode).In CI, using all available CPUs performed better than using only half. Nevertheless, I would prefer to use the more conservative value of
numCpus / 2for the auto mode. This still gives a speedup of more than 2.