Encapsulation of polyglot/java Dependencies

[radeusgd]

With the imports/exports discussion we are talking about encapsulation of out private modules.

Many of our libraries rely on Java helpers that are imported using the polyglot java import statements. To achieve proper encapsulation of library internals, we should ensure that these internal helper libraries (std-base, std-table etc.) are private and not accessible from user projects.

High level idea

For most cases, we want to ensure that a JAR file that is in a polyglot directory of a given project can only be imported from within that project and not from other places.

Re-exporting

We still want some helpers to be visible in other modules. For example, some helpers from std-table (the Table library companion JAR), are also used in the Database library. To do so we can re-export such symbols by wrapping them in Enso wrappers. This will be enough for helper methods, but it may not be enough for types.

For the time being this should be enough for our usecases.
Potentially, for longer-term, maybe we should devise a better control over where the JARs are importable from. Maybe we can include some information in the package.yaml file or have a sibling directory public-polyglot - where such JARs would be visible to the outside world?

Implementation

Currently, loading a JAR is achieved by TruffleLanguage::Env::addToHostClassPath. This does not have the control for imports we need. We would probably need a check at compiler level which verifies if a given polyglot java import clause is allowed based on the contents of the polyglot directory associated with the project from which the compiled Enso file comes from.

This may require some non-trivial logic to scan each JAR added to the classpath and for each path it adds to the classpath, we would need to associate it with the project it comes from. Then the compiler can verify for each polyglot java import if the current project has the 'permission' to access a given path.

Collisions

This still leaves the question of - what if two libraries contain JARs in their polyglot directories that have clashing classpaths? For example, two versions of the same Java library? Ideally, due to the encapsulation each library should be able to import each own version, but that would require some stricter separation of classpaths.

Until we have some 'clever' solution for this, we may want to forbid importing clashing libraries. Ideally we should detect if adding a new library to the classpath will shadow another one and disallow importing such a library.

Alternatives

The Java Module System seems to be solving a similar task. Maybe we can somehow plug into it and exploit it instead of inventing our own wheels? It partially depends on if that is possible to do in how Truffle/Graal handles the modules for the polyglot interface.

Tasks

Preview Give feedback

1. Remove workaround from https://github.com/enso-org/enso/pull/9664/commits/60b38f491853a7ed02e64debfbb730a36ee9a029

[JaroslavTulach]

Isolation can be achieved by using different ClassLoader for each library. E.g. having different HostClassLoader for each library.

[JaroslavTulach]

The implementation of encapsulation is provided by #10714. It uses Java Platform Module System and defines a ModuleLayer for each library with JARs in polyglot/java directory.

[enso-bot]

Jaroslav Tulach reports a new STANDUP for yesterday (2024-07-30):

Progress: - #7082: ClassPath & Using ModuleLayer for loading classes: #10714

• attempt to fix tests: Package libraries Java code as Java Modules #10714 (comment) It should be finished by 2024-08-05.

[enso-bot]

Jaroslav Tulach reports a new STANDUP for today (2024-07-31):

Progress: - Turning libraries into modules by Pavel: 7e865ce#r1697852981

• uses/provides: c4c1842#r1697927708
• removal of add_to_class_path
• turning std-table into Java module
• pair IGV with Pavel
• meeting James
• introducing Base_Internal_Tests: 74fa9e1
• standups & meetings It should be finished by 2024-08-05.

[enso-bot]

Jaroslav Tulach reports a new STANDUP for yesterday (2024-08-01):

Progress: - libs as Java Modules

• Warnings & IGV: 9823d2c#r1699474211
• package.yaml requires: https://github.com/enso-org/enso/pull/10714/files#r1699702548
• more of requires: Package libraries Java code as Java Modules #10714 (comment)
• attempt to create demo project It should be finished by 2024-08-05.

GitHub

Package libraries Java code as Java Modules by JaroslavTulach · Pull Request #10714 · enso-org/enso

Pull Request Description Implements #7082 by creating a module layer per library. Removes the obsolete add_to_class_path concept. Important Notes

Checklist
Please ensure that the following checkli...

[enso-bot]

Jaroslav Tulach reports a new STANDUP for yesterday (2024-08-02):

Progress: - case study ModuleLayer project: https://github.com/JaroslavTulach/multilayerstudy

• case study finished: https://github.com/JaroslavTulach/multilayerstudy/blob/8fb71ce75914865c68f3b057086342de8ffd03d1/README.md
• review parser changes: Improve parser contextualization #10734 (review)
• review Decimal changes: Mixed Decimal/Float operations throw error or attach warning #10725 (review)
• standup It should be finished by 2024-08-05.

GitHub

GitHub - JaroslavTulach/multilayerstudy: Case study of using modules and module layers

Case study of using modules and module layers. Contribute to JaroslavTulach/multilayerstudy development by creating an account on GitHub.

GitHub

multilayerstudy/README.md at 8fb71ce75914865c68f3b057086342de8ffd03d1 · JaroslavTulach/multilayerstudy

Case study of using modules and module layers. Contribute to JaroslavTulach/multilayerstudy development by creating an account on GitHub.

[enso-bot]

Jaroslav Tulach reports a new STANDUP for yesterday (2024-08-03):

Progress: - WarningBenchmarks: #10555 (comment)

• requesting changes for dump IR: Add org.enso.compiler.dumpIr system prop #10740 (review)
• found usage of getUncached: WithWarnings uses EnsoHashMap to speed things up #10555 (comment)
• arguing about syntax: Improve parser contextualization #10734 (comment) It should be finished by 2024-08-05.

[JaroslavTulach]

@gunnarmorling provided a reference to layrry - A Launcher and API for Modularized Java Applications.

[enso-bot]

Jaroslav Tulach reports a new STANDUP for yesterday (2024-08-05):

Progress: - Deep review of #10725 (review)

• Polishing multilayerstudy: https://github.com/JaroslavTulach/multilayerstudy/blob/master/README.md
• got a reference to layrry: https://x.com/gunnarmorling/status/1820327297825570868
• REPL import all: Import all available libraries in --repl mode #10746
• standups, (demo) meetings It should be finished by 2024-08-05.

GitHub

multilayerstudy/README.md at master · JaroslavTulach/multilayerstudy

Case study of using modules and module layers. Contribute to JaroslavTulach/multilayerstudy development by creating an account on GitHub.

X (formerly Twitter)

x.com

[enso-bot]

Jaroslav Tulach reports a new STANDUP for yesterday (2024-08-06):

Progress: - REPL script verified: 78db664

• reviews like Various fixes to visualizations #10745 (review)
• Warning & DataflowError PR: Attaching a warning to error is no op #10765
• REPL at the end PR: Show REPL console at the end of main #10767 It should be finished by 2024-08-12.

[enso-bot]

Jaroslav Tulach reports a new STANDUP for yesterday (2024-08-07):

Progress: - #10736: error&warnings PR cleanup: #10765 (comment)

• reviewed and approved warnings via EnsoHashMap: WithWarnings uses EnsoHashMap to speed things up #10555 (review)
• all libs in --repl merged: Import all available libraries in --repl mode #10746
• standup & meetings It should be finished by 2024-08-12.

[enso-bot]

Jaroslav Tulach reports a new STANDUP for yesterday (2024-08-08):

Progress: - Reimplemented: #10765 (comment)

• launcher TODO list: https://github.com/orgs/enso-org/discussions/10121#discussioncomment-10272101
• Outdated launcher: Detect outdated launcher #10779 It should be finished by 2024-08-12.

GitHub

Launching Enso programs instantly · enso-org · Discussion #10121

Enso is dynamically compiled language built on top of JVM - e.g. GraalVM - and its Truffle framework as such programs written in Enso can run pretty fast - once the dynamic compiler does its job. T...

[enso-bot]

Jaroslav Tulach reports a new STANDUP for yesterday (2024-08-09):

Progress: - review of revert: #10778

• analyzing failures: Detect outdated launcher #10779 (comment)
• standup
• merged: Attaching a warning to error is no op #10765
• investigating failure on MacOSX It should be finished by 2024-08-12.

[enso-bot]

Jaroslav Tulach reports a new STANDUP for yesterday (2024-08-12):

Progress: - review: #10793 (review)

• standup
• investigating Ydoc-server: Ydoc native image #10783 (comment) It should be finished by 2024-08-12.

[JaroslavTulach]

• the integration of Provide engine-runner & language-server as a separate JPMS modules #10157 allows this work to continue

[JaroslavTulach]

Now we are blocked by poor integration of native-image with ServiceLoader & JPMS modules. The NativeImageModularDemo demonstrates the problem. The bug has been reported to GraalVM team:

• [Native Image] Missing support for ServiceLoader services registered via module-info oracle/graal#9952
• Update on Nov 25, 2024: being implemented by @ivan-ristovic

Dec, 2024 Update:

The fix has been merged (oracle/graal#10202) and will be part of GraalVM 24.2.

We will also backport to GraalVM 23.1.

[JaroslavTulach]

Native Image support for ServiceLoader is implemented and will be part of next GraalVM

• Register services in package.yaml instead via ServiceLoader #11868 might be a better alternative to overcome the problems of ServiceLoader
• because it also solves Native Image run includes SPI for libraries that are not loaded, leading to crashes #11707
• we don't need https://github.com/JaroslavTulach/multilayerstudy tricks to make ServiceLoader work with multiple layers