Replace language server WebSocket connection with Ydoc #14438

4e6 · 2025-12-08T10:02:53Z

Pull Request Description

close Avoid networking layer of Akka by ydoc-server talking directly to the engine+ls #14142

Important Notes

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

The documentation has been updated, if necessary.
Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
All code follows the
Scala,
Java,
TypeScript,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
Unit tests have been written where possible.
If meaningful changes were made to logic or tests affecting Enso Cloud integration in the libraries,
or the Snowflake database integration, a run of the Extra Tests has been scheduled.
- If applicable, it is suggested to paste a link to a successful run of the Extra Tests.

JaroslavTulach · 2025-12-09T09:28:25Z

lib/java/ydoc-api/src/main/java/module-info.java

@@ -0,0 +1,3 @@
+module org.enso.ydoc.api {
+  exports org.enso.ydoc.api;


There is no need for a new API module.

there already is YdocServerApi

please put the new interfaces there as inner classes of YdocServerApi.

Possibly you can move YdocServerApi to the new org.enso.ydoc.api module.

but the whole API needs to be co-located

JaroslavTulach · 2025-12-09T09:30:54Z

lib/java/ydoc-api/src/main/java/org/enso/ydoc/api/NoOpMessageCallbacks.java

+package org.enso.ydoc.api;
+
+public class NoOpMessageCallbacks implements MessageCallbacks {
+  public static final NoOpMessageCallbacks INSTANCE = new NoOpMessageCallbacks();


Please don't put NoOpMessagesCallbacks implementation class into the API

remove it

if you have to have it, then encapsulate it

put public static final MessageCallbacks NO_OP into MessageCallbacks and make the implementation class package private

JaroslavTulach

ideally HostAccess is configured internally
and both tests and production code just work with the default configuration
possible trick to achieve that is to crate a proxy around interfaces passed to JavaScript

JaroslavTulach · 2025-12-10T06:30:19Z

lib/java/ydoc-server/src/main/java/org/enso/ydoc/server/Ydoc.java

      return this;
    }

+    public Builder hostAccessBuilder(HostAccess.Builder hostAccessBuilder) {


Why is this a builder?

Isn't just HostAccess instance good enough?

Suggested change

public Builder hostAccessBuilder(HostAccess.Builder hostAccessBuilder) {

public Builder hostAccess(HostAccess hostAccessBuilder) {

without WebEnvironment.defaultHostAccess being public, one might need a builder

but having both builder and WebEnvironment.defaultHostAccess seems unnecessary

JaroslavTulach · 2025-12-10T06:35:46Z

lib/java/ydoc-server/src/main/java/org/enso/ydoc/server/Main.java

+          WebEnvironment.defaultHostAccess
+              // allowImplementations is required to call methods on JS objects from Java, i.e. to
+              // call methods on `YjsChannel` object returned from JS
+              .allowImplementations(YjsChannel.class)


allowing implementation of a well known interface is OK

but opening up allowAccess to a random callbacks.getClass() class is a security flaw

please create a proxy around callbacks and expose its methods

preferrably via @HostAccess.Explicit

or via allowAccess(ProxyMessageCallbacks.getDeclararedMethod("")) but while hardcoding ProxyMessageCallbacks class

The current state with:

.allowImplementations(YjsChannel.class) .allowPublicAccess(true);

is certainly better than the original one. Thanks.

Still, it be good to remove .allowPublicAccess(true) and use @HostAccess.Explicit annotation instead.

JaroslavTulach · 2025-12-10T06:36:49Z

lib/java/ydoc-api/src/main/java/module-info.java

@@ -0,0 +1,3 @@
+module org.enso.ydoc.api {
+  exports org.enso.ydoc.api;


Possibly you can move YdocServerApi to the new org.enso.ydoc.api module.

but the whole API needs to be co-located

lib/scala/json-rpc-server/src/main/scala/org/enso/jsonrpc/YdocJsonRpcServer.scala

JaroslavTulach · 2025-12-15T06:05:10Z

app/ydoc-channel/src/YjsChannel.ts

+          for (const item of items) {
+            // Only notify handlers if the message is from another sender
+            if (item.senderId !== this.senderId) {
+              this.notifyHandlers(item.payload)


As of 8713e6f we are probably dealing with ever growing Y.Array

somehow we need to find out that the "message has been delivered" to all handlers

that's kind of hard in a distributed framework

where handlers can appear and diappear

one solution would be to add receiverId to item

deliver the message only when senderId matches receiverId

e.g. if (item.receiverId === this.senderId)`

if there was only one receiver of a message

then it'd be the receiver's responsibility to remove the item from array

…ruffleString

JaroslavTulach

It all looks quite decent. Few comments inline.

JaroslavTulach · 2025-12-29T12:45:28Z

app/ydoc-channel/src/YjsChannel.ts

+/**
+ * A bidirectional communication channel backed by Y.Array.
+ *
+ * This class allows multiple parties to send and receive messages through a shared


really "multiple parties"?

shouldn't the YjsChannel be just bidirectional?

JaroslavTulach · 2025-12-29T13:42:09Z

app/ydoc-shared/src/util/net/__tests__/YjsTransport.test.ts

+
+// Polyfill DOM event types for Node.js environment
+if (typeof globalThis.CloseEvent === 'undefined') {
+  ;(globalThis as any).CloseEvent = class CloseEvent extends Event {


Line prefixed by ;?

Suggested change

;(globalThis as any).CloseEvent = class CloseEvent extends Event {

(globalThis as any).CloseEvent = class CloseEvent extends Event {

JaroslavTulach · 2025-12-29T13:46:20Z

engine/runner-common/src/main/java/org/enso/runner/common/YdocServerApi.java

 import java.util.ServiceLoader;
+import org.enso.ydoc.api.MessageCallbacks;

 public abstract class YdocServerApi {


Why don't we put YdocServerApi into org.enso.ydoc.api module as well? As this comment suggests?

JaroslavTulach · 2025-12-29T13:58:46Z

lib/java/ydoc-polyfill/src/test/java/org/enso/ydoc/api/CallbacksTest.java

+    @Override
+    public void onConnect(YjsChannel channel) {


Suggested change

@Override

public void onConnect(YjsChannel channel) {

@Override

@HostAccess.Export

public void onConnect(YjsChannel channel) {

JaroslavTulach · 2025-12-29T13:59:32Z

lib/java/ydoc-polyfill/src/test/java/org/enso/ydoc/api/CallbacksTest.java

+            // public access is required to recognize Java lambdas passed to
+            // `YjsChannel::subscribe` method as JS functions.
+            .allowPublicAccess(true)
+            .allowAccess(TestCallbacks.class.getDeclaredMethod("onConnect", YjsChannel.class))


Suggested change

.allowAccess(TestCallbacks.class.getDeclaredMethod("onConnect", YjsChannel.class))

// no need for TestCallbacks registration

rather use @HostAccess.Explicit

JaroslavTulach · 2025-12-29T14:00:52Z

lib/java/ydoc-polyfill/src/test/java/org/enso/ydoc/api/CallbacksTest.java

+
+  @Test
+  public void onConnectSend() throws Exception {
+    var res = new AtomicReference<>();


I'd slightly prefer you to create CountingTest class and register its instance instead

annotate its set method with @HostAccess.Explicit

JaroslavTulach · 2025-12-29T14:02:47Z

lib/java/ydoc-server/src/main/java/org/enso/ydoc/server/Main.java

        "javax.management.**;java.lang.**;java.rmi.**;javax.security.auth.Subject;!*");

    if (args.length == 2) {
      var then = System.currentTimeMillis();


Is this public static void main still useful for anything?

Shouldn't it be removed?

JaroslavTulach · 2025-12-29T14:04:30Z

lib/java/ydoc-server/src/main/java/org/enso/ydoc/server/Main.java

+          WebEnvironment.defaultHostAccess
+              // allowImplementations is required to call methods on JS objects from Java, i.e. to
+              // call methods on `YjsChannel` object returned from JS
+              .allowImplementations(YjsChannel.class)


The current state with:

.allowImplementations(YjsChannel.class) .allowPublicAccess(true);

is certainly better than the original one. Thanks.

Still, it be good to remove .allowPublicAccess(true) and use @HostAccess.Explicit annotation instead.

JaroslavTulach · 2025-12-29T14:11:18Z

lib/java/ydoc-server/src/main/java/org/enso/ydoc/server/Ydoc.java

                  var bindings = ctx.getBindings("js");
                  bindings.putMember("YDOC_HOST", hostname);
                  bindings.putMember("YDOC_PORT", port);
+                  bindings.putMember("YDOC_MESSAGE_CALLBACKS", callbacks);


registering non-primitive type as global symbol seems strange

with YDOC_HOST one could post it as environment variable, but

one cannot register callbacks via environment variable

why don't create a initializeYdocServer function

and call it here directly with proper arguments?

JaroslavTulach · 2025-12-29T14:14:40Z

app/ydoc-server-polyglot/src/main.ts


 configureAllDebugLogs(debug)

+if (YDOC_MESSAGE_CALLBACKS == undefined) {


this check makes main.ts unusable from command line

at least I believe usage from CLI was the motivation for these YDOC_ (environment) variables

why don't we define:

function initializeYdocServer(YDOC_HOST, YDOC_PORT, YDOC_LS_DEBUG, YDOC_MESSAGE_CALLBACKS) {

and then just call this method with proper arguments from Java?

4e6 self-assigned this Dec 8, 2025

4e6 added the CI: No changelog needed Do not require a changelog entry for this PR. label Dec 8, 2025

4e6 force-pushed the wip/db/14142-ydoc-websocket branch 2 times, most recently from 7356274 to 4371781 Compare December 8, 2025 18:03

JaroslavTulach reviewed Dec 9, 2025

View reviewed changes

JaroslavTulach reviewed Dec 10, 2025

View reviewed changes

4e6 force-pushed the wip/db/14142-ydoc-websocket branch from 71e9b86 to 8713e6f Compare December 13, 2025 16:18

JaroslavTulach reviewed Dec 15, 2025

View reviewed changes

lib/scala/json-rpc-server/src/main/scala/org/enso/jsonrpc/YdocJsonRpcServer.scala Show resolved Hide resolved

JaroslavTulach reviewed Dec 15, 2025

View reviewed changes

4e6 force-pushed the wip/db/14142-ydoc-websocket branch from 486cb1d to c1ee0ea Compare December 22, 2025 19:58

4e6 and others added 19 commits December 29, 2025 11:46

add: ydoc-api

0c0989e

add: ydoc-channel

6acb057

update: simplify JsonRpcServer

76cc37d

add: YdocJsonRpcServer

963bc71

add: YjsWSTransport

20a5c3f

misc: format

b44c382

add: JsonRpcServer callbacks

3b64da0

add: callbacks test

409b850

refactor: yjs transport

90f22bb

DRAFT: yjs transport

63b9823

DRAFT: connect gui rpc

ceccc84

DRAFT: yjs callbacks

f559cc4

misc: fixes

e449afd

Test simulating Ydoc's YjsChannel communication

7d99b24

Test that yields: No persistance for com.oracle.truffle.api.strings.T…

7a2d53b

…ruffleString

Need TruffleString Persistance to interact with JavaScript

ff92e83

DRAFT: yjs transport tests

da7830a

DRAFT: debug

8424c31

DRAFT: backend transport

2355986

4e6 added 6 commits December 29, 2025 11:47

update: working yjs transport

3166266

feat: pass executor

3a70a9c

update: cleanup internal YjsChannel array

c8075a0

test: fix callbacks test

040c8f0

feat: syncronized callbacks

87f17c3

revert: pass executor

f62447f

4e6 force-pushed the wip/db/14142-ydoc-websocket branch from f122b95 to 9625b1f Compare December 29, 2025 11:49

4e6 added 2 commits December 29, 2025 11:49

misc: format

9625b1f

feat: YjsChannel WebSocket events

9002cfe

JaroslavTulach reviewed Dec 29, 2025

View reviewed changes

feat: YjsDataChannel

7c15531

enso-bot bot mentioned this pull request Dec 30, 2025

Run Standard.Snowflake in dual JVM mode #14474

Open

4 tasks

4e6 added 2 commits December 30, 2025 13:55

feat: binary channel callbacks

74763c2

update: yjs handle ArrayBuffer messages

5baf1a1

4e6 force-pushed the wip/db/14142-ydoc-websocket branch from 27df1af to b5f7639 Compare January 1, 2026 20:50

update: yjs callbacks handle Uint8Arrays

b5f7639

		@@ -0,0 +1,3 @@
		module org.enso.ydoc.api {
		exports org.enso.ydoc.api;

	public Builder hostAccessBuilder(HostAccess.Builder hostAccessBuilder) {
	public Builder hostAccess(HostAccess hostAccessBuilder) {

	;(globalThis as any).CloseEvent = class CloseEvent extends Event {
	(globalThis as any).CloseEvent = class CloseEvent extends Event {

	.allowAccess(TestCallbacks.class.getDeclaredMethod("onConnect", YjsChannel.class))
	// no need for TestCallbacks registration


		configureAllDebugLogs(debug)

		if (YDOC_MESSAGE_CALLBACKS == undefined) {

Replace language server WebSocket connection with Ydoc #14438

Are you sure you want to change the base?

Replace language server WebSocket connection with Ydoc #14438

Conversation

4e6 commented Dec 8, 2025

Pull Request Description

Important Notes

Checklist

Uh oh!

JaroslavTulach Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JaroslavTulach Dec 9, 2025 •

edited

Loading

JaroslavTulach Dec 29, 2025 •

edited

Loading