Describe converting MRC images to OME-NGFF (zarr)

[blowekamp]

[blowekamp]

The bioformats2raw tool is recommended for direct converting the mrc files to the OME-NGFF zarr pyramid format supported by Neuroglancer.

Note neuroglancer support signed 16 and 8 bit pixel types in zarr so that conversion is also no longer needed.

The following is the recommended command line for 2D MRC files:

bioformats2raw --scale-format-string '%2$d' --downsample-type AREA --compression=blosc --compression-properties cname=zstd --compression-properties clevel=5 --compression-properties shuffle=1 --tile_width 512 --tile_height 512  input.mrc output.zarr

The compression properties were selected based to the reported performance of images of noise from this paper. . This choice follows the recommendation of the conclusion.

edit: updated the tile recommendation to be 512x512. If the pixel type is 16-bit that would make each chunk 512KB

[blowekamp]

The spec for OME-NGFF is here:
https://ngff.openmicroscopy.org/latest/

[blowekamp]

For 3D Tomogram MRC files the input will need to converted with bioformats2raw. For 3D visualization in neuroglancer the zarr chunks should be 3D (XYZ) to allow for smooth scrolling and cross section visualization. These chunks should be about 1MB is size to be efficiently downloaded and processed in neuroglancer. The bioformats2raw only resamples in XY.
The following configures bioformats2raw to only produce the full resolution zarr array. This is then processed with zarr_build_multiscales to generate the multi scale images/arrays.

The tomograms pixels are mostly in XY, and the recommended chunk size is 256,256,64 (XYZ). The zarr_build_multiscales will reuse the same chunk size as the full resolution image, and shrink the dimensions in XYZ.

bioformats2raw --scale-format-string '%2$d' --compression=blosc --compression-properties cname=zstd --compression-properties clevel=5 --compression-properties shuffle=1  --resolutions 1 --chunk_depth 64 --tile_width 256 --tile_height 256 input.mrc output.zarr
zarr_build_multiscales output.zarr

The chunk size could be optimized depending on the size of the minimum dimension to ensure the chunk size if ~1MB.

edit: update to use build multiscales

[blowekamp]

For 3D FIBSIM MRC files the input will need to converted with bioformats2raw. The current pixel type is int8, which has been tested to work with the following command line arguments and with Neuroglancer for visualization.

For the same reasons for 3D tomograms zarr_build_multiscales will be used.

The FIBSIM number of pixels and be large in XYZ, and the current recommendation for isotropic chunking of 128,128,128.

bioformats2raw --scale-format-string '%2$d' --compression=blosc --compression-properties cname=zstd --compression-properties clevel=5 --compression-properties shuffle=1  --resolutions 1 --chunk_depth 128 --tile_width 128 --tile_height 128 input.mrc output.zarr
zarr_build_multiscales output.zarr

The chunk size could be optimized depending on the size of the minimum dimension to ensure the chunk size if ~1MB. Also note that the above only differs from the tomogram command by the recommended chunk size.

[blowekamp]

A note on parrallization and multithreading. These processes seem to scale well up to 8 CPUs, and some improvements with 16 on sample data sets. The custom pytools commands such as zarr_build_multiscales by default will use the local CPU and multithreading to parallize to the full number of CPUS/cores available.

bioformats2raw defaults to 4 cores, and does not autodetect the system. The number of threads to be used can be explicitly specified with the max_workers command line argument. e.g. --max_workers 16

[blowekamp]

A note on mrc_visual_min_max. It has been updated to support zarr arrays. The sub-directory of the zarr array needs to be passed as an argument. For example if out.zarr is created then the following command line processes the highest resolution array mrc_visual_min_max out.zarr/0 --mad 3