Exchange dataset input with `tiled_viewer` component

[Wiebke]

Currently, the input dataset selection in the application is implemented by populating a dropdown with a list of options assembled by iterating over elements in the container specified by DATA_TILED_URI. For any element, the element is added if:

• it is an array, or
• it is a container that contains a single array (occurs when Tiled has grouped sequence of images in a folder)
• it is a container with a recognized Spec (e.g., NXtomoproc) that indicates an array exists further down the hierarchy (used for nested HDF5/Nexus structures).

mlex_highres_segmentation/utils/data_utils.py

Lines 135 to 164 in 354dc33

	def get_data_sequence_by_name(self, project_name):
	"""
	Data sequences may be given directly inside the main client container,
	but can also be additionally encapsulated in a folder, multiple container or in a .nxs file.
	We make use of specs to figure out the path to the 3d data.
	"""
	if self.data_client is None or project_name is None:
	return None
	project_client = self.data_client[project_name]
	# If the project directly points to an array, directly return it
	if isinstance(project_client, ArrayClient):
	return project_client
	# If project_name points to a container
	elif isinstance(project_client, Container):
	# Check if the specs give us information about which sub-container to access
	specs = project_client.specs
	# TODO: Read yaml file with spec to path mapping
	if any(spec.name == "NXtomoproc" for spec in specs):
	# Example for how to access data if the project container corresponds to a
	# nexus-file following the NXtomoproc definition
	# TODO: This assumes that a validator has checked the file on ingestion
	# Otherwise we should first test if the path holds data
	return project_client["entry/data/data"]
	# Enter the container and return first element
	# if it represents an array
	if len(list(project_client)) == 1:
	sequence_client = project_client.values()[0]
	if isinstance(sequence_client, ArrayClient):
	return sequence_client
	return None

Assume the configured dataset container (e.g. DATA_TILED_URI=https://tiled-seg.als.lbl.gov/api/v1/metadata/reconstruction) has the following structure:

/reconstruction/
/reconstruction/dataset1                   ← array
/reconstruction/dataset2                   ← container
/reconstruction/dataset2/dataset2_         ← array
/reconstruction/dataset3                   ← has `Spec(name='NXtomoproc') → array under entry/data/data

In this case, the dropdown would contain [dataset1, dataset2, dataset3] as options. These dropdown entries are later used as project names when referencing derived data (e.g. masks, segmentations) and are used to determine where results are stored under MASK_TILED_URI and SEG_TILED_URI. In case of new additions to Tiled, the options can be updated through a refresh button.

This approach is functional for small catalogs but scales poorly and restricts dataset selection to a fixed hierarchy depth.

We need this more flexible and robust in case of large catalogs, and also support various authentication mechanism for Tiled access. This can be accomplished with integration of the tiled_viewer Dash component which wraps a corresponding React Tiled Viewer component. This would allow interactive, lazy-loaded navigation through large catalogs and support authenticated access.

[Wiebke]

In the current (draft) implementation, dataset selection triggers a server-side callback that retrieves data from Tiled.
The corresponding high-level sequence diagram is shown below.

sequenceDiagram
  autonumber
  participant B as Mlex Highres<br/>(Client / Browser)

  box Docker Network
    participant M as Mlex Highres<br/>(Server)
    participant T as Tiled<br/>(Data Service)
  end

  Note over B,T: Browse available datasets
  B->>T: GET DATA_TILED_URI/search/?sort=-
  T-->>B: 200 JSON (list of containers)

  Note over B,M: User selects a dataset
  B->>M: POST /_dash-update-component (selection info)

  Note over M,T: Server fetches data from Tiled
  M->>T: GET DATA_TILED_URI/path_of_selected_array
  T-->>M: 200 array/data

  Note over M,B: Dash callback result updates the component
  M-->>B: 200 JSON (updated figure / layout)

Loading

We could convert this to a client-side callback (browser fetches arrays directly from Tiled), but that doesn’t solve some caveats I am running into in various deployments: Both the browser and the server need Tiled access, but are operating in different network contexts.

Browser: has no knowledge of internal container addresses, and hence should use a host-mapped or publicly accessible URL, e.g. http://localhost:8000/api/v1/metadata/reconstruction/path_of_selected_array or https://tiled-seg.als.lbl.gov/api/v1/metadata/reconstruction/path_of_selected_array
Services in Docker (Mlex Highres, Model Code): previously used exclusively the internal Docker hostname (or a publicly accessible Tiled instance) for both selection and data access, e.g. http://tiled:8000/api/v1/metadata/reconstruction/path_of_selected_array

In a local deployment where all services are running in Docker (which also includes NGINX which configures access to Tiled under the / route) I am observing the following (?api_key=... is omitted):

Source	Address	Result	Notes
Browser (address bar)	https://machinename.als.lbl.gov/api/v1/	✅	Works
	http://localhost/api/v1	✅	Works
Browser (MLEX Highres)	https://machinename.als.lbl.gov/api/v1/	❌	Blocked by CORS
	http://localhost/api/v1	❌	Blocked by CORS
Container	http://tiled:8000/api/v1	✅	Works
	https://machinename.als.lbl.gov/api/v1/	❌	IP resolves, after that any requests time out

Example browser console error for CORS:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost/api/v1/search/&sort=-. (Reason: CORS request did not succeed). Status code: (null).

For reaching the hostname from inside a container, I assume firewall configuration and/or additional Docker settings are needed to allow reaching the same machine via its public hostname.