Add WebSocket module and streaming query evaluation to exist-core

[joewiz]

Summary

Adds WebSocket infrastructure to exist-core with two endpoints:

• /ws — Channel-based pub/sub messaging (ported from monex) with _monitor query lifecycle broadcasting
• /ws/eval — Real-time streaming XQuery evaluation with cancellation, progress reporting, and timing breakdown

This is the foundation for removing Java code from monex, enabling future LSP wire protocol support, and making eXist's interactive tools feel modern and responsive.

Based on: PR #6144 (Jetty 12 upgrade) — should merge after that PR.

/ws — Channel Pub/Sub

Ported from monex's RemoteConsoleEndpoint into a general-purpose WebSocket module.

XQuery ws:send("channel", $data)
    → WebSocketModule → WebSocketAdapter → WebSocketEndpoint → Browser

Clients connect, subscribe to channels via {"channel": "name"}, and receive JSON messages. A 500ms heartbeat ping keeps connections alive through proxies.

_monitor Channel (new)

Admin clients can subscribe to the _monitor channel to receive real-time query lifecycle events:

• Event-driven: started, progress, completed, error, cancelled events from /ws/eval queries
• Periodic snapshots: All running queries (including REST/XQueryServlet) via ProcessMonitor, broadcast every 1 second

XQuery Functions

Function	Signature	Purpose
ws:log	($items as item()*) as empty-sequence()	Log to "default" channel
ws:log	($channel, $items)	Log to specific channel
ws:send	($channel, $items)	Send JSON to channel
ws:broadcast	($items as item()*)	Send to ALL clients
ws:channel-count	($channel) as xs:integer	Count subscribers

/ws/eval — Streaming Query Evaluation (new)

A WebSocket endpoint for real-time XQuery evaluation — the XQuery equivalent of a modern database's interactive query console protocol.

Protocol (JSON over WebSocket)

Client → Server:

Action	Purpose
eval	Execute query with optional streaming, variables, serialization, timeout
cancel	Cancel a running query by ID
compile	Compile-check query without execution (syntax validation)
admin-cancel	DBA-only: kill any running query via ProcessMonitor

Server → Client:

Type	Purpose
progress	Phase updates: parsing → compiling → evaluating → serializing
result	Serialized data chunks with more/done indicators and timing
error	XPath error with code, message, line, column
cancelled	Confirmation with items-produced count
compile	Success/failure with diagnostics array

Key Features

• Streaming: Configurable chunk size (default 1000 items). Large results stream incrementally — the client sees data as it's produced.
• Cancellation: cancel action sets a flag on XQueryWatchDog. Works mid-evaluation and mid-serialization.
• Timing breakdown: Parse, compile, evaluate, serialize phases measured independently.
• Authentication: Basic auth on WebSocket handshake, falls back to guest.
• Admin-cancel: DBA users can kill any running query (same as system:kill-running-xquery).
• Progress reporting: Phase transitions broadcast to both the eval client and _monitor subscribers.

Example

const ws = new WebSocket('ws://localhost:8080/exist/ws/eval');

ws.send(JSON.stringify({
    action: 'eval',
    id: 'q-1',
    query: 'for $i in 1 to 1000000 return $i',
    "chunk-size": 1000,
    serialization: { method: 'adaptive' }
}));

ws.onmessage = (event) => {
    const msg = JSON.parse(event.data);
    if (msg.type === 'result') appendToOutput(msg.data);
    if (msg.type === 'progress') updateStatusBar(msg.phase);
    if (!msg.more) showTiming(msg.timing);
};

// Cancel
ws.send(JSON.stringify({ action: 'cancel', id: 'q-1' }));

New Classes

Class	Package	Purpose
WebSocketEndpoint	o.e.xquery.functions.websocket	@ServerEndpoint("/ws") — channel pub/sub, heartbeat
WebSocketModule	o.e.xquery.functions.websocket	XQuery module (ws: namespace)
ConsoleCompatModule	o.e.xquery.functions.websocket	Backward-compatible console: namespace
EvalWebSocketEndpoint	o.e.http.ws	@ServerEndpoint("/ws/eval") — streaming eval
EvalSession	o.e.http.ws	Per-connection state, query cancellation
QueryExecutor	o.e.http.ws	XQuery execution with streaming, progress, timing
EvalProtocol	o.e.http.ws	JSON message parsing and generation
QueryMonitorBroadcaster	o.e.http.ws	Bridges eval lifecycle → _monitor channel

Backward Compatibility

The console: namespace (http://exist-db.org/xquery/console) is provided by ConsoleCompatModule, which delegates to WebSocketModule. Existing monex XQuery code using console:log() works unchanged.

Test plan

WebSocket pub/sub (3 tests):

• Connect and receive heartbeat ping
• Subscribe to channel and receive message
• Channel count reflects subscribers

Eval endpoint (20 tests):

• Simple eval (1 + 1 → "2")
• Variables (external variable binding with type casting)
• Compile error (syntax error returns error message)
• Compile action (success and failure with diagnostics)
• Streaming results (500 items, chunk-size 100 → 5 chunks)
• Large result streaming (100K items, chunk-size 1000 → 100 chunks)
• Cancellation (long query cancelled mid-flight)
• Rapid cancel (cancel before compilation completes)
• Max execution time (timeout enforcement via watchdog)
• Timing (result includes parse/compile/eval/serialize breakdown)
• Progress reporting (phase messages during execution)
• Serialization options (adaptive method)
• Binary result handling (xs:base64Binary)
• Map/array serialization (adaptive output)
• Module-load-path resolution (import module from /db)
• Admin-cancel permission check (non-DBA gets denied)
• Monitor channel (_monitor receives query lifecycle events)
• Connection cleanup (disconnect mid-query releases resources)
• Error recovery (invalid JSON doesn't kill connection)
• Invalid message / missing query handling

Total: 23 integration tests

What This Enables

• eXide: Stream results live, cancel runaway queries, see timing breakdown
• Sandbox: Notebook cells with progress bars and incremental output
• monex: Live query dashboard via _monitor channel
• VS Code: "Run Query" with streaming results and cancel button
• xst CLI: xst eval --stream prints results as they arrive

CI Status

Integration tests pass on macOS and ubuntu. Known CI issues:

• ubuntu unit tests: Timeout — WebSocket integration tests add overhead to the already tight CI time limit
• windows integration: Flaky WebSocket test failures due to platform-specific timing (passes locally)
• Container image: Build failure (inherits from Jetty 12 base — merge Upgrade Jetty 11.0.25 to 12.0.16 (EE10) #6144 first)
• Codacy: Pre-existing style issues — no new findings from this PR

🤖 Generated with Claude Code