Skip to content

Commit ebaa6fe

Browse files
committed
[SPARK-30572][BUILD] Add a fallback Maven repository
### What changes were proposed in this pull request? This PR aims to add a fallback Maven repository when a mirror to `central` fail. ### Why are the changes needed? We use `Google Maven Central` in GitHub Action as a mirror of `central`. However, `Google Maven Central` sometimes doesn't have newly published artifacts and there is no guarantee when we get the newly published artifacts. By duplicating `Maven Central` with a new ID, we can add a fallback Maven repository which is not mirrored by `Google Maven Central`. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Manually testing with the new `Twitter` chill artifacts by switching `chill.version` from `0.9.3` to `0.9.5`. ``` $ rm -rf ~/.m2/repository/com/twitter/chill* $ mvn compile | grep chill Downloading from google-maven-central: https://maven-central.storage-download.googleapis.com/repos/central/data/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom Downloading from central_without_mirror: https://repo.maven.apache.org/maven2/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom Downloaded from central_without_mirror: https://repo.maven.apache.org/maven2/com/twitter/chill_2.12/0.9.5/chill_2.12-0.9.5.pom (2.8 kB at 11 kB/s) ``` Closes #27281 from dongjoon-hyun/SPARK-30572. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit c992716) Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent fbd95b1 commit ebaa6fe

File tree

2 files changed

+23
-0
lines changed

2 files changed

+23
-0
lines changed

.github/workflows/branch-2.4.yml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,13 @@ jobs:
5151
export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"
5252
export MAVEN_CLI_OPTS="--no-transfer-progress"
5353
mkdir -p ~/.m2
54+
# `Maven Central` is too flaky in terms of downloading artifacts in `GitHub Action` environment.
55+
# `Google Maven Central Mirror` is too slow in terms of sycing upstream. To get the best combination,
56+
# 1) we set `Google Maven Central` as a mirror of `central` in `GitHub Action` environment only.
57+
# 2) we duplicates `Maven Central` in pom.xml with ID `central_without_mirror`.
58+
# In other words, in GitHub Action environment, `central` is mirrored by `Google Maven Central` first.
59+
# If `Google Maven Central` doesn't provide the artifact due to its slowness, `central_without_mirror` will be used.
60+
# Note that we aim to achieve the above while keeping the existing behavior of non-`GitHub Action` environment unchanged.
5461
echo "<settings><mirrors><mirror><id>google-maven-central</id><name>GCS Maven Central mirror</name><url>https://maven-central.storage-download.googleapis.com/repos/central/data/</url><mirrorOf>central</mirrorOf></mirror></mirrors></settings>" > ~/.m2/settings.xml
5562
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pmesos -Pkubernetes -Phive -Phive-thriftserver -Pscala-${{ matrix.scala }} -P${{ matrix.hadoop }} -Phadoop-cloud install
5663
rm -rf ~/.m2/repository/org/apache/spark

pom.xml

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -242,6 +242,22 @@
242242
<enabled>false</enabled>
243243
</snapshots>
244244
</repository>
245+
<repository>
246+
<id>central_without_mirror</id>
247+
<!--
248+
This is used as a fallback when a mirror to `central` fail.
249+
For example, when we use Google Maven Central in GitHub Action as a mirror of `central`,
250+
this will be used when Google Maven Central is out of sync due to its late sync cycle.
251+
-->
252+
<name>Maven Repository</name>
253+
<url>https://repo.maven.apache.org/maven2</url>
254+
<releases>
255+
<enabled>true</enabled>
256+
</releases>
257+
<snapshots>
258+
<enabled>false</enabled>
259+
</snapshots>
260+
</repository>
245261
</repositories>
246262
<pluginRepositories>
247263
<pluginRepository>

0 commit comments

Comments
 (0)