SPEC: specify openjdk-devel package as dependency #5071

amaslenn · 2020-04-23T11:41:13Z

What

Specify openjdk-devel package as a build dependency.

Why ?

For some reason, Fedora doesn't pull openjdk-devel as a dependency for Maven, have to set it manually.

For some reason Fedora doesn't pull openjdk-devel as a dependency for Maven, have to set it manually.

amaslenn · 2020-04-23T11:42:25Z

@yosefe would be nice to have it for v1.8.1 release along with #5070.

ucx.spec.in

yosefe · 2020-05-04T08:33:30Z

/azp run

azure-pipelines · 2020-05-04T08:33:41Z

Azure Pipelines successfully started running 1 pipeline(s).

yosefe · 2020-05-04T08:46:41Z

https://dev.azure.com/ucfconsort/ucx/_build/results?buildId=5869&view=logs&jobId=1b6e0548-13f3-5bbf-46b7-02377d74d406&j=1b6e0548-13f3-5bbf-46b7-02377d74d406&t=a3dbc0da-a4bc-5096-251c-c5e42b51def6

2020-05-04T08:37:23.0954673Z + rpmbuild -bs '--define=_rpmdir /__w/1/s/rpm-dist' '--define=_srcrpmdir /__w/1/s/rpm-dist' '--define=_sourcedir /__w/1/s' '--define=_specdir /__w/1/s' '--define=_builddir /__w/1/s' --buildroot=/__w/1/s/_rpm ucx.spec
2020-05-04T08:37:23.2180138Z Wrote: /__w/1/s/rpm-dist/ucx-1.9.0-1.fc30.src.rpm
2020-05-04T08:37:23.2844667Z + rpmbuild -bb '--define=_rpmdir /__w/1/s/rpm-dist' '--define=_srcrpmdir /__w/1/s/rpm-dist' '--define=_sourcedir /__w/1/s' '--define=_specdir /__w/1/s' '--define=_builddir /__w/1/s' --buildroot=/__w/1/s/_rpm ucx.spec --with cma --without cuda --without gdrcopy --with ib --without ib_cm --without knem --with rdmacm --without rocm --without ugni --without xpmem --with java
2020-05-04T08:37:23.3033504Z error: Failed build dependencies:
2020-05-04T08:37:23.3035036Z 	java-latest-openjdk-devel >= 8 is needed by ucx-1.9.0-1.fc30.x86_64
2020-05-04T08:37:23.3100668Z 
2020-05-04T08:37:23.3181336Z ##[error]Bash exited with code '1'.
2020-05-04T08:37:23.3396988Z ##[section]Finishing: Build tarball

petro-rudenko · 2020-05-04T09:35:44Z

I think it's irrelevant now as of merging #5095
Maven already takes java-latest-openjdk-devel.
java-latest-openjdk-devel has no version since it's latest. Or to remove >= 8 from this PR

yosefe · 2020-05-04T15:15:53Z

@amaslenn shall we close this?

amaslenn · 2020-05-05T06:55:34Z

This is still relevant.

Here is how to install openjdk-devel in Fedora:

# dnf search '*openjdk*' | grep devel
Last metadata expiration check: 0:00:58 ago on Mon May  4 15:39:46 2020.
java-openjdk-devel.i686 : Renamed to java-latest-openjdk-devel                                                                              java-openjdk-devel.x86_64 : Renamed to java-latest-openjdk-devel

I need to update UCX fedora Docker image to have this package installed and the checks for this PR will pass.

@yosefe would you prefer a separate PR for Fedora Docker update or do it here?

amaslenn · 2020-05-05T06:57:39Z

I think it's irrelevant now as of merging #5095
Maven already takes java-latest-openjdk-devel.
java-latest-openjdk-devel has no version since it's latest. Or to remove >= 8 from this PR

openjdk-devel is not installed as a Maven dependency in Fedora environment, so we have to explicitly specify it.

yosefe · 2020-05-05T08:18:58Z

@yosefe would you prefer a separate PR for Fedora Docker update or do it here?

In this PR

amaslenn · 2020-05-06T07:01:01Z

@petro-rudenko could you please take a look?

List of installed Java packages:

$ docker run ucfconsort.azurecr.io/ucx/fedora:2 rpm -qa | grep java
java-1.8.0-openjdk-headless-1.8.0.242.b08-0.fc30.x86_64
tzdata-java-2019c-2.fc30.noarch
javapackages-tools-5.3.0-4.fc30.noarch
java-latest-openjdk-14.0.0.36-3.rolling.fc30.x86_64
java-latest-openjdk-devel-14.0.0.36-3.rolling.fc30.x86_64
java-latest-openjdk-headless-14.0.0.36-3.rolling.fc30.x86_64
javapackages-filesystem-5.3.0-4.fc30.noarch

Error:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project jucx: Compilation failure
[ERROR] No compiler is provided in this environment. Perhaps you are running on a JRE rather than a JDK?
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
make[5]: *** [Makefile:992: native_headers.stamp] Error 1

petro-rudenko · 2020-05-18T10:32:43Z

Clean fedora:30 image:

 dnf install maven
======================================================================================================================================
 Package                                      Arch        Version                                          Repository            Size
======================================================================================================================================
Installing:
 maven                                        noarch      1:3.5.4-5.module_f30+8611+513ae82a               updates-modular       11 k
Installing dependencies:
...
 java-1.8.0-openjdk                           x86_64      1:1.8.0.252.b09-0.fc30                           updates              255 k
 java-1.8.0-openjdk-headless                  x86_64      1:1.8.0.252.b09-0.fc30                           updates               33 M
...
Installing weak dependencies:
 gtk2                                         x86_64      2.24.32-6.fc30                                   updates              3.5 M
 java-1.8.0-openjdk-devel                     x86_64      1:1.8.0.252.b09-0.fc30                           updates              9.7 M

Then you remove only java-1.8.0-openjdk-devel. And installs java-latest-openjdk-devel. Maven still finds java8. Any case it won't compile with java14 yet, even when it'll find correct java14 compiler, will fix this soon.

petro-rudenko · 2020-05-18T10:57:41Z

So basically it won't compile on JDK-14 for java8 bytecode (because we use internal API to get byte buffer address):
https://stackoverflow.com/a/55605311/3271168

Solution is the following:

Compile on java8 (lowest supported compiler, as we do). No need this PR then
Compile on java14 for and produce code for java11+ (not good, since a lot of users still uses java8 runtime.

amaslenn · 2020-05-18T11:06:45Z

@petro-rudenko that means we should specify java-1.8.0-openjdk-devel as a dependency, right? Otherwise, it won't be installed on modern systems.

petro-rudenko · 2020-05-18T11:15:18Z

@amaslenn maven brings it on fedora30 container

petro-rudenko · 2020-05-18T11:17:43Z

We compiled with java9. Seems only java14 doesn't allow to use internal API, in favor of : https://www.baeldung.com/java-foreign-memory-access

amaslenn · 2020-05-18T11:29:47Z

@amaslenn maven brings it on fedora30 container

it is not on fedora@rawhide (faced it when tried to release v1.8.0)

petro-rudenko · 2020-05-18T11:32:49Z

Then let's install java-1.8.0-openjdk-devel except of java-latest-openjdk-devel

amaslenn · 2020-05-20T06:27:45Z

@yosefe f30 got gcc/cppcheck updated, v9.1.1 --> v9.3.1 and 1.88 --> 1.90 accordingly. They found new issues in code, could you please take a look?

buildlib/azure-pipelines-pr.yml

amaslenn · 2020-05-21T12:41:24Z

@petro-rudenko I'm getting the following issue when building UCX in Fedora/Koji env:

mvn -B -T 1C -f /builddir/build/BUILD/ucx-1.9.0/bindings/java/pom.xml -Dmaven.repo.local=./build-java/.deps -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn compile
[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[ERROR] Unresolveable build extension: Plugin org.sonatype.plugins:nexus-staging-maven-plugin:1.6.8 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.sonatype.plugins:nexus-staging-maven-plugin:jar:1.6.8 @ 
 @ 
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]   
[ERROR]   The project org.openucx:jucx:1.9.0 (/builddir/build/BUILD/ucx-1.9.0/bindings/java/pom.xml) has 1 error
[ERROR]     Unresolveable build extension: Plugin org.sonatype.plugins:nexus-staging-maven-plugin:1.6.8 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.sonatype.plugins:nexus-staging-maven-plugin:jar:1.6.8: Could not transfer artifact org.sonatype.plugins:nexus-staging-maven-plugin:pom:1.6.8 from/to gcs-maven-central-mirror (https://maven-central.storage-download.googleapis.com/maven2/): transfer failed for https://maven-central.storage-download.googleapis.com/maven2/org/sonatype/plugins/nexus-staging-maven-plugin/1.6.8/nexus-staging-maven-plugin-1.6.8.pom: Unknown host maven-central.storage-download.googleapis.com: Name or service not known -> [Help 2]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/PluginManagerException
make[3]: *** [Makefile:992: native_headers.stamp] Error 1
make[3]: Leaving directory '/builddir/build/BUILD/ucx-1.9.0/bindings/java/src/main/native'
make[2]: *** [Makefile:506: all-recursive] Error 1
make[2]: Leaving directory '/builddir/build/BUILD/ucx-1.9.0/bindings/java'
make[1]: *** [Makefile:730: all-recursive] Error 1
make[1]: Leaving directory '/builddir/build/BUILD/ucx-1.9.0'

Full log: https://kojipkgs.fedoraproject.org//work/tasks/7768/44767768/build.log

petro-rudenko · 2020-05-24T10:51:29Z

Is there internet connection in this container?

amaslenn · 2020-05-24T13:11:39Z

Is there internet connection in this container?

Not sure, but it somehow pulls the sources and other packages.

amaslenn · 2020-05-28T08:33:04Z

@petro-rudenko ping

amaslenn · 2020-05-31T14:18:02Z

@yosefe turned out our current approach for packaging JUCX is invalid: java package should be a separate rpm. Also, we have to use Fedora's rpm macroses for Maven and this will require splitting UCX and JUCX builds.

According to @petro-rudenko JUCX users don't need rpms, they use Maven. I suggest removing the current JUCX packaging instructions and configure UCX with --without-java SPEC file.

If you think we need to package it anyway, let's discuss the priority: implementation will take time, it is hard to estimate right now. And it blocks v1.8.* release for Fedora.

yosefe · 2020-05-31T14:19:38Z

According to @petro-rudenko JUCX users don't need rpms, they use Maven. I suggest removing the current JUCX packaging instructions and configure UCX with --without-java SPEC file.

+1 - let's build UCX RPM without java

SPEC: specify openjdk-devel package as dependency

54088e8

For some reason Fedora doesn't pull openjdk-devel as a dependency for Maven, have to set it manually.

amaslenn requested review from petro-rudenko and yosefe April 23, 2020 11:41

hoopoepg approved these changes Apr 23, 2020

View reviewed changes

petro-rudenko approved these changes Apr 23, 2020

View reviewed changes

yosefe reviewed Apr 24, 2020

View reviewed changes

ucx.spec.in Show resolved Hide resolved

yosefe added the Waiting for Author Response label Apr 25, 2020

petro-rudenko mentioned this pull request Apr 27, 2020

JUCX: use javac path for JAVA_HOME #5093

Closed

RPMSPEC: use min required jdk version instead of hardcoded

7902a39

petro-rudenko mentioned this pull request Apr 27, 2020

JUCX: detect javac to use for JAVA_HOME #5095

Merged

Andrey Maslennikov added 2 commits May 6, 2020 09:46

CI: add java-latest-openjdk-devel to fedora docker

3706141

CI: use fedora:2 docker image

d74e1f1

amaslenn force-pushed the topic/rpm-spec-jdk branch from 8033c81 to d74e1f1 Compare May 6, 2020 06:46

CI/SPEC: use jdk v1.8.0 as JUCX dependency

4a893ba

yosefe reviewed May 20, 2020

View reviewed changes

buildlib/azure-pipelines-pr.yml Show resolved Hide resolved

yosefe approved these changes May 21, 2020

View reviewed changes

yosefe merged commit c875d29 into openucx:master May 21, 2020

amaslenn deleted the topic/rpm-spec-jdk branch May 21, 2020 12:10

amaslenn mentioned this pull request May 31, 2020

RPM: package UCX without Java #5224

Merged

SPEC: specify openjdk-devel package as dependency #5071

SPEC: specify openjdk-devel package as dependency #5071

Uh oh!

Conversation

amaslenn commented Apr 23, 2020

What

Why ?

Uh oh!

amaslenn commented Apr 23, 2020

Uh oh!

Uh oh!

yosefe commented May 4, 2020

Uh oh!

azure-pipelines bot commented May 4, 2020

Uh oh!

yosefe commented May 4, 2020

Uh oh!

petro-rudenko commented May 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yosefe commented May 4, 2020

Uh oh!

amaslenn commented May 5, 2020

Uh oh!

amaslenn commented May 5, 2020

Uh oh!

yosefe commented May 5, 2020

Uh oh!

amaslenn commented May 6, 2020

Uh oh!

petro-rudenko commented May 18, 2020

Uh oh!

petro-rudenko commented May 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amaslenn commented May 18, 2020

Uh oh!

petro-rudenko commented May 18, 2020

Uh oh!

petro-rudenko commented May 18, 2020

Uh oh!

amaslenn commented May 18, 2020

Uh oh!

petro-rudenko commented May 18, 2020

Uh oh!

amaslenn commented May 20, 2020

Uh oh!

Uh oh!

amaslenn commented May 21, 2020

Uh oh!

petro-rudenko commented May 24, 2020

Uh oh!

amaslenn commented May 24, 2020

Uh oh!

amaslenn commented May 28, 2020

Uh oh!

amaslenn commented May 31, 2020

Uh oh!

yosefe commented May 31, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

petro-rudenko commented May 4, 2020 •

edited

Loading

petro-rudenko commented May 18, 2020 •

edited

Loading