Native libraries are searched in lib/polyglot project dir #11874
Conversation
Akirathan
added
the
CI: Clean build required
Akirathan
changed the title
|
It was a mistake to try to create a different Polyglot_Utils was loaded by a HostClassLoader for Base pkg. Let's just stick with a single HostClassLoader that will override findLibrary.
Reverting in 1649ddf and 05851f5
|
So that no extraction is done unnecessarily.
Size of |
| assert dir.exists() && dir.isDirectory(); | ||
| nativeLibPaths.add(dir.getAbsolutePath()); | ||
| var current = System.getProperty("java.library.path"); | ||
| RuntimeSystemProperties.register( |
There was a problem hiding this comment.
I don't think this is necessary neither good idea.
Please remember that NI build runs in build time, while the .so, .dll files are about to be located in runtime. The paths are going to be different.
Can you tell me: What was failing and why you think this change is helpful? We don't need anything like this to load .so for enso_parser...
There was a problem hiding this comment.
For enso_parser, we have full control over explicit search for the library in Parser$Worker$initializeLibraries - in that method, we try to explicitly load the enso_parser native library first with System.loadLibrary, and then with System.load.
For opencv, we don't have full control. Its native library opencv_java470 is loaded in the static initializer of nu.pattern.OpenCV with System.loadLibrary("opencv_java470"). OpenCV class is used by classes in our std-bits/image. In JVM mode, all classes from std-bits/image (classes in package org.enso.image) are loaded by HostClassLoader, therefore, also OpenCV class is loaded by HostClassLoader, and so, System.loadLibrary("opencv_java470") delegates to HostClassLoader.findLibrary. In native image, there is no HostClassLoader, and we have to deal with the System.loadLibrary call inside OpenCV differently.
The solution to include path to the native library inside RuntimeSystemProperties from NI build time was the only solution I could think of. It points to a path like distribution/lib/Standard/Image/0.0.0-dev/polyglot/lib/... where the native libraries are located. All the tests are currently passing, because we test NI only via cmdline, and we don't try to package the NI inside the AppImage. It is definitely not the most robust solution. I am opened to any alternatives.
TL;DR; Is there any other way how to hook into System.loadLibrary calls during runtime in NI other than setting java.library.path via RuntimeSystemProperties during NI build?
There was a problem hiding this comment.
In native image, there is no HostClassLoader,
Really? How comes there is no HostClassLoader? Of course it cannot load any new classes, but the classloader can/should still be there...
There was a problem hiding this comment.
This is an example of the use of HostClassLoader. Still some problems there, but this way we should be able to get HostClassLoader "into the game".
There was a problem hiding this comment.
I have tried multiple times to somehow get HostClassLoader "into the game", but failed after many attempts. See #11874 (comment) . The alternative solution mentioned in #11874 (comment) is much easier and seems to work. It does not modify configuration of NI at all.
There was a problem hiding this comment.
Copying the libraries seems like a reasonable and portable fix. Let's use it for now and keep the findLibrary approach for later (for example #7082).
JaroslavTulach
left a comment
JaroslavTulach
left a comment
There was a problem hiding this comment.
The build time vs. runtime classpath issue has to be solved.
| assert dir.exists() && dir.isDirectory(); | ||
| nativeLibPaths.add(dir.getAbsolutePath()); | ||
| var current = System.getProperty("java.library.path"); | ||
| RuntimeSystemProperties.register( |
There was a problem hiding this comment.
This is an example of the use of HostClassLoader. Still some problems there, but this way we should be able to get HostClassLoader "into the game".
|
After many failed attempts to somehow enforce loading of native libraries via
|
|
Different idea to try: Done in e6956db |
JaroslavTulach
left a comment
JaroslavTulach
left a comment
There was a problem hiding this comment.
I am glad to see the copying of native libs next to enso binary working. It avoids hard coding the paths of the build machine and seems like a good solution for now.
| NativeLibraryFinder.listAllNativeLibraries(pkg, FileSystem$.MODULE$.defaultFs()); | ||
| for (var nativeLib : nativeLibs) { | ||
| var out = new File(nativeLibDir, nativeLib.getName()); | ||
| Files.copy(nativeLib.toPath(), out.toPath()); |
There was a problem hiding this comment.
This line should override the library if it exists:
--- engine/runner/src/main/java/org/enso/runner/EnsoLibraryFeature.java
+++ engine/runner/src/main/java/org/enso/runner/EnsoLibraryFeature.java
@@ -1,12 +1,12 @@
package org.enso.runner;
import static scala.jdk.javaapi.CollectionConverters.asJava;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Path;
+import java.nio.file.StandardCopyOption;
import java.util.LinkedHashSet;
import java.util.TreeSet;
import org.enso.compiler.core.EnsoParser;
import org.enso.compiler.core.ir.module.scope.imports.Polyglot;
import org.enso.filesystem.FileSystem$;
@@ -110,7 +112,7 @@ public final class EnsoLibraryFeature implements Feature {
NativeLibraryFinder.listAllNativeLibraries(pkg, FileSystem$.MODULE$.defaultFs());
for (var nativeLib : nativeLibs) {
var out = new File(nativeLibDir, nativeLib.getName());
- Files.copy(nativeLib.toPath(), out.toPath());
+ Files.copy(nativeLib.toPath(), out.toPath(), StandardCopyOption.REPLACE_EXISTING);
nativeLibPaths.add(out.getAbsolutePath());
}
}Removes the copying of native libraries next to the generated NI binary. Native libraries of packages are loaded from the same location in both JVM and AOT modes. **Old behavior:** [EnsoLibraryFeature](https://github.com/enso-org/enso/blob/7106fdd1cc32902554e7318e811d66e66cc6f3e4/engine/runner/src/main/java/org/enso/runner/EnsoLibraryFeature.java) copies all the native libraries within `**/polyglot/lib/**` next to the generated NI binary. The motivation for that is mentioned in the description of #11874. **New behavior:** [EnsoLibraryFeature](https://github.com/enso-org/enso/blob/7106fdd1cc32902554e7318e811d66e66cc6f3e4/engine/runner/src/main/java/org/enso/runner/EnsoLibraryFeature.java) no longer copies native libraries. Instead, once a package is loaded in [DefaultPackageRepository.registerPackageInternal](https://github.com/enso-org/enso/blob/d369fcbe69f3eb1baa5fe7a5b8d24354b74067ad/engine/runtime/src/main/scala/org/enso/interpreter/runtime/DefaultPackageRepository.scala#L203-L247), paths to its native libraries are added to the native library search path via [NativeLibrarySearchPath](https://github.com/enso-org/enso/blob/9d8866944b145add67bbb944328ded1e3c4a3bba/engine/runtime/src/main/java/org/enso/interpreter/runtime/nativeimage/NativeLibrarySearchPath.java). Inspired by https://github.com/Akirathan/native-lib-loader. # Important Notes - In JVM mode, native libraries of packages are still loaded by [HostClassLoader.findLibrary](https://github.com/enso-org/enso/blob/7658faf64563ef25e8f04003892cfb074fd907c5/engine/runtime/src/main/java/org/enso/interpreter/runtime/HostClassLoader.java#L86-L109). - In AOT mode, [com.oracle.svm.core.jdk.NativeLibraries#usrPaths](https://github.com/oracle/graal/blob/91061cb6a151ffaa83e05f6852384537c2c85e6b/substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/jdk/NativeLibraries.java#L67) field is changed at runtime.
Fixes #11483
Pull Request Description
Native libraries used via JNI from polyglot Java code can be located in
polyglot/libdirectories in projects. New docs is inenso/docs/polyglot/java.md
Lines 68 to 93 in a9c60a5
Important Notes
Standard.Imagedepends onopencv.jarwhich contains all native libraries for all platforms:All native libraries have 352 MB, but we only need a single one. For example:
In this PR, sbt extracts all the native libraries from
opencv.jarand puts them intoStandard/Image/.../polyglot/libdirectory. In subsequent PRs, we may want to drop all the native libraries for different platforms.Native image building modification
EnsoLibraryFeature called during native image build scans all the native libraries inside std libraries, and copies them next to the generated executable. That way, NI will find the library without
HostClassLoaderor without changingjava.library.pathsystem property.Checklist
Please ensure that the following checklist has been satisfied before submitting the PR:
Scala,
Java,
TypeScript,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
or the Snowflake database integration, a run of the Extra Tests has been scheduled.