HiCT JVM Implementation

Launching pre-built version

For users of .jar distribution

This section is intended for bioinformatics users who download a ready-to-run fat JAR from GitHub Releases. You need to install Java 21+ (this project is built for Java 21 bytecode). Download the latest fat JAR from the Releases page (Assets section). NOTE: prebuilt native bundles are currently provided for Windows (tested on 10/11) and Linux with glibc (common Debian/Ubuntu-like distributions). Alpine/musl is not supported by these bundled binaries. Current prebuilt artifacts are AMD64-only. On Windows you might need to install Microsoft Visual C++ Redistributable.

Quick start

1. Download the latest -fat.jar from the Releases page (Assets) and rename it to hict-fat.jar.

2. Place your .hict.hdf5, .mcool, .cool, .agp, and .fasta files under a single directory. If you have .hic files from Juicebox, you can convert them using hictk into Cooler format (use the finest resolution) and then use cooler to build zoom pyramid and compute balance weights, for example:

hictk convert arabiensis.hic arabiensis.1kb.cool --resolutions 1kbp
cooler zoomify -n 8 -r 4DN --balance --balance-args '--nproc 8 --ignore-diags 3' -o arabiensis.hic.mcool arabiensis.1kb.cool

Next you can either convert that file into HiCT format through the WebUI (File → Convert Coolers), or continue in CLI:

java -jar hict-fat.jar convert mcool-to-hict --input='arabiensis.hic.mcool' --output='arabiensis.hic.mcool.hict.hdf5' --compression-algorithm=DEFLATE --compression=6 --parallelism=-1

1. To start HiCT Server with WebUI execute the following command:

   java -jar hict-fat.jar start-server

   Directory with files is set using DATA_DIR environment variable, by default it scans subtree of the directory in which hict-fat.jar is launched from. In Linux you may set it as follows:

   DATA_DIR=/path/to/data/ java -jar hict-fat.jar start-server

2. Open WebUI at http://localhost:8080.

CLI commands (summary)

# API + WebUI (default mode, includes converters in WebUI as descibed below)
java -jar hict-fat.jar start-server

# API only (no WebUI)
java -jar hict-fat.jar start-api-server

# Convert .mcool -> .hict.hdf5 (CLI mode)
java -jar hict-fat.jar convert mcool-to-hict \
  --input /data/sample.mcool \
  --output /data/sample.hict.hdf5

# Convert .hict.hdf5 -> .mcool (CLI mode)
java -jar hict-fat.jar convert hict-to-mcool \
  --input /data/sample.hict.hdf5 \
  --output /data/sample.mcool

Get full CLI help:

java -jar hict-fat.jar --help
java -jar hict-fat.jar start-server --help
java -jar hict-fat.jar start-api-server --help
java -jar hict-fat.jar convert --help
java -jar hict-fat.jar convert mcool-to-hict --help
java -jar hict-fat.jar convert hict-to-mcool --help

WebUI conversion (Experimental / W.I.P.)

Warning

WebUI conversion is experimental and may be slower or less stable than the CLI.

1. Open the WebUI.

2. Use File → Convert Coolers.

3. Track progress in the conversion window.

API access (Experimental / W.I.P.)

Warning

The API is still evolving. Endpoints, parameters, and response formats may change.

Example (Python) for fetching a submatrix tile as an image:

import requests

host = "http://localhost:5000"
params = {
    "version": 0,
    "bpResolution": 10000,
    "format": "PNG_BY_PIXELS",
    "row": 0,
    "col": 0,
    "rows": 512,
    "cols": 512,
}

r = requests.get(f"{host}/get_tile", params=params)
r.raise_for_status()
data = r.json()
png_data_url = data["image"]
print(png_data_url[:64])

To apply visualization/normalization settings before fetching tiles:

• POST /set_visualization_options with visualization parameters.

• POST /set_normalization with normalization settings.

• Then call /get_tile as shown above.

Supported platforms / JDK details

• OS/CPU (prebuilt libs): Linux (glibc) and Windows, AMD64.

• Not bundled by default: macOS variants and Linux ARM variants.

• JDK: Java 19 or newer is required for running/building this repository.

Startup options and CLI

The fat JAR is runnable and exposes a CLI with subcommands:

• start-server — API + WebUI (default when no args are given)

• start-api-server — API only (no WebUI)

• convert — conversion tools

  • convert mcool-to-hict

  • convert hict-to-mcool

Help:

java -jar hict.jar --help
java -jar hict.jar convert --help
java -jar hict.jar convert mcool-to-hict --help

Environment variables supported by the server startup:

• DATA_DIR — directory that is scanned recursively for .hict.hdf5, .agp, fasta, .cool, and .mcool files.

• VXPORT — API gateway port, default 5000.

• WEBUI_PORT — WebUI port, default 8080.

• SERVE_WEBUI — true/false, default true.

• TILE_SIZE — default visualization tile size, default 256.

• MIN_DS_POOL / MAX_DS_POOL — min/max pool sizes used when opening chunked datasets.

Launch examples (fat JAR)

Linux (bash)

DATA_DIR=/home/${USER}/hict/data java -jar hict.jar

# API only
DATA_DIR=/home/${USER}/hict/data java -jar hict.jar start-api-server

# Explicit server (API + WebUI)
DATA_DIR=/home/${USER}/hict/data java -jar hict.jar start-server

Windows (cmd)

set DATA_DIR="D:\hict\data"
set WEBUI_PORT="8888"
java -jar hict.jar start-server

Windows (PowerShell)

$env:DATA_DIR = "D:\hict\data"
$env:WEBUI_PORT = "8888"
java -jar hict.jar start-server

Custom JVM options

DATA_DIR=/home/${USER}/hict/data java -ea -Xms512M -Xmx16G -jar hict.jar start-api-server

Launch examples (Gradle, from source)

# Default: runs HiCT CLI (equivalent to `java -jar ...`)
./gradlew clean run

# Explicit modes
./gradlew run --args="start-server"
./gradlew run --args="start-api-server"

Converter workflows (.mcool ↔ .hict.hdf5)

CLI commands

Use the JVM CLI for both directions:

# mcool -> hict
java -jar hict.jar convert mcool-to-hict \
  --input /data/sample.mcool \
  --output /data/sample.hict.hdf5

# hict -> mcool
java -jar hict.jar convert hict-to-mcool \
  --input /data/sample.hict.hdf5 \
  --output /data/sample.roundtrip.mcool

Web conversion API flow

Typical asynchronous conversion sequence used by WebUI/integrations:

1. Upload: POST /api/convert/upload

   • Upload source file and target format metadata.

   • Response returns a jobId.

2. Status polling: GET /api/convert/status/{jobId}

   • Poll until state becomes DONE or FAILED.

3. Download: GET /api/convert/download/{jobId}

   • Download converted artifact when status is DONE.

Recommended size limits:

• Keep upload limits explicit at ingress/proxy and app gateway.

• For JVM safety, avoid unbounded request bodies in production; set max request size and timeouts.

• For very large matrices, prefer direct local file conversion (CLI) and then load resulting artifacts through DATA_DIR.

Scaffolding API behavior notes

Scaffolding operations are served as POST endpoints and return updated assembly information:

• /reverse_selection_range

• /move_selection_range

• /split_contig_at_bin

• /group_contigs_into_scaffold

• /ungroup_contigs_from_scaffold

• /move_selection_to_debris

Important tile-version expectation:

• Tile requests use GET /get_tile?…​&version=<n>.

• If the requested version is older than server-side tile version, server returns HTTP 204 (no tile body) to force client invalidation.

• If the requested version is newer, server advances the internal version counter.

• Practical client rule: after each scaffolding mutation, increment your tile version and refresh visible tile requests.

Startup errors and JHDF5 native library troubleshooting

During startup, you may see several native-library load attempts with warnings/errors. This can be expected because different platform-specific library names are tried.

If startup completes and API/WebUI are healthy, these warnings can be non-fatal.

When native loading actually fails:

1. Confirm architecture match (AMD64 JVM + AMD64 native bundle).

2. Confirm OS compatibility (Linux glibc; not Alpine/musl).

3. On Linux, ensure native/plugin paths are discoverable, for example:

   export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/path/to/hdf5/lib:/path/to/hdf5/lib/plugin"
   export HDF5_PLUGIN_PATH="/path/to/hdf5/lib/plugin"

4. On Windows, install/update Visual C++ runtime redistributables.

5. Verify Java version (java -version) is 19+.

6. If tiles fail to render but server starts, inspect logs for UnsatisfiedLinkError and HDF5 plugin load failures.

Production checklist (short)

Before deploying to production, verify:

• Logging: structured logs, retention, and centralized collection.

• Metrics/health: request latency/error metrics and liveness/readiness checks.

• Limits: request body size, timeouts, and JVM heap sizing are set explicitly.

• Graceful shutdown: stop accepting traffic, finish in-flight requests, then terminate.

• Backup/cleanup: regular backup strategy for source/converted files and periodic cleanup of temporary/intermediate artifacts.

Building HiCT_JVM from source

To build from source:

./gradlew clean build

Dependency management workflow

This project uses Gradle dependency locking (gradle.lockfile) to keep transitive dependency resolution reproducible.

• Refresh lock state after dependency changes:

  ./gradlew dependencies --write-locks

• Inspect the resolved version for a specific dependency before/after updates:

  ./gradlew dependencyInsight --dependency org.slf4j:slf4j-api --configuration runtimeClasspath
  ./gradlew dependencyInsight --dependency ch.qos.logback:logback-classic --configuration runtimeClasspath
  ./gradlew dependencyInsight --dependency org.jetbrains:annotations --configuration compileClasspath

Commit both build.gradle.kts and gradle.lockfile together whenever lock state changes.

Current progress on modifying HDF5 and JHDF5 configuration resides in my personal repository. Modified configuration is necessary to rebuild native libraries (HDF5, HDF5 plugins and JHDF5 should all be build as dynamic libraries). However, prebuilt native libraries for AMD64 Windows and Linux platforms are already present in HiCT_JVM repository. Missing platforms are Linux on armv7 and aarch64 and MacOS (both amd64 and aarch64 variants).

Conversion tools (CLI + API)

A native converter module is now available in JVM codebase with two services:

• McoolToHictConverter (mcool-to-hict)

• HictToMcoolConverter (hict-to-mcool)

CLI launcher:

./gradlew runConversionCli --args="convert hict-to-mcool --input=/data/sample.hict.hdf5 --output=/data/sample.mcool --resolutions=10000,50000 --compression=4 --chunk-size=8192"
./gradlew runConversionCli --args="convert mcool-to-hict --input=/data/sample.mcool --output=/data/sample.hict.hdf5 --resolutions=10000,50000 --parallelism=16"

Arguments:

• --input=<path> source file path

• --output=<path> destination file path

• --resolutions=<comma-separated> optional resolution filter

• --compression=<0..9> deflate level (0 means chunked/no deflate)

• --chunk-size=<N> chunk size for streaming traversal

• --agp=<file.agp> --apply-agp apply AGP before hict-to-mcool export

• --parallelism=<N> max worker threads (default: available CPU cores)

Web API endpoints:

• POST /convert/upload (multipart + query params: direction, resolutions, compression, chunkSize, applyAgp, agpPath, parallelism)

• GET /convert/jobs/:jobId

• GET /convert/download/:jobId

Conversion jobs are asynchronous, include streaming logs/error details, enforce upload size limit and have temporary file cleanup TTL.