idr0001-graml-sysgro to NGFF

[will-moore]

idr0001 has 192 x 96-Well Plates, 6 acquisitions each.
1 Plate converted below is 47 GB.
Approx 4.5 TB in total.
bioformats2raw took ~30mins to convert 1 Plate => approx 4 days in total.

NB: The need to convert multi-acquisition Flex data (idr0001) is because the support for that hasn't been ported from IDR to mainline BioFormats: ome/bioformats#3537

[will-moore]

convert first plate for testing...
on pilot-zarr1-dev

~/bioformats2raw-0.6.0-24/bin/bioformats2raw "/uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/JL_120731_S6A/Meas_01(2012-07-31_10-41-12)/001001001.flex" JL_120731_S6A
...
2024-01-08 13:36:06,658 [main] WARN  loci.formats.FormatHandler - mismatch between image count, names and factors (count=36864, names=32, factors=32)
2024-01-08 13:36:06,661 [main] WARN  loci.formats.FormatHandler - mismatch between image count, names and factors (count=36864, names=32, factors=32)
2024-01-08 13:36:06,664 [main] WARN  loci.formats.FormatHandler - mismatch between image count, names and factors (count=36864, names=32, factors=32)
2024-01-08 13:36:06,668 [main] WARN  loci.formats.FormatHandler - mismatch between image count, names and factors (count=36864, names=32, factors=32)
2024-01-08 13:36:06,671 [main] WARN  loci.formats.FormatHandler - mismatch between image count, names and factors (count=36864, names=32, factors=32)
...

(at 2pm)...

Done about 14:15 (approx 45 mins).

[will-moore]

make bucket etc...
then:

(base) [wmoore@pilot-zarr1-dev ~]$ ./mc cp -r /data/idr0001/JL_120731_S6A.ome.zarr uk1s3/idr0001/zarr
...0731_S6A.ome.zarr/OME/METADATA.ome.xml: 96.57 GiB / 96.57 GiB ━━━━━━━━━━━━━━ 48.81 MiB/s 33m45s

[will-moore]

Looks like we are getting duplicates of each of the 6 acquisitions for each Plate.
We would expect each Well to have 6 Fields but we get 12 (6 pairs of duplicates):
https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/idr0001/zarr/JL_120731_S6A.ome.zarr/A/1/

[sbesson]

There are a series of issues we might need to review here. For reference, the source data corresponding to the example above is https://ftp.ebi.ac.uk/pub/databases/IDR/idr0001-graml-sysgro/20151116-verified/JL_120731_S6A/.

As can be seen above, there are 6 measurement folders, which are interpreted as plate acquisitions using the IDR Flex reader. Looking at the file listing for each measurement, there are 2 fields of views per well and plate acquisition.

001001001.flex
001001002.flex
001002001.flex
001002002.flex
...

In IDR, each well of each plate acquisition only contains 1 field per view and the fileset only includes the Flex files ending with 001.flex. I cannot comment know whether this is to be treated as an issue or an active decision made during the loading of this historical study /cc @joshmoore

If the intent is to create a Zarr dataset matching the current IDR representation, I think of two options:

• either point bioformats2raw at a secondary structure where only the ...001.flex files are present
• oruse omero-cli-zarr against live IDR

[will-moore]

Let's try omero-cli-zarr export for comparison...
On idr-ftp...
Update to latest ome/omero-cli-zarr#147

conda activate omero_zarr_export
pip uninstall omero-cli-zarr

# cloned https://github.com/ome/omero-cli-zarr/pull/147 and merged https://github.com/ome/omero-cli-zarr/pull/156
$ cd /data/idr0001/omero_cli_zarr/omero-cli-zarr
$ pip install -e .

$ omero zarr export Plate:2552 --name_by name

...export status... 5 out of 96 Wells done in 35 mins. 7 mins per Well is 11 hours per Plate!

... completed at 22:08 - ~11.5 hours.

Upload...

./mc cp -r /data/idr0001/omero_cli_zarr/JL_120731_S6B.ome.zarr uk1s3/idr0001/zarr

Looks good at https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr0001/zarr/JL_120731_S6B.ome.zarr/

[will-moore]

Tried exporting Polygons as labels... but this fails as the Polygons are overlapping:

(omero_zarr_export) [wmoore@pilot-zarr1-dev omero_cli_zarr]$ omero zarr polygons Plate:2552 --source-image=JL_120731_S6B.ome.zarr

Found 51 mask shapes in 51 ROIs
Unique dimensions: {'T': {None}, 'C': {None}, 'Z': {None}}
source_image JL_120731_S6B.ome.zarr/D/7/0
Ignoring dimensions {'C', 'T', 'Z'}
Traceback (most recent call last):
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/bin/omero", line 11, in <module>
    sys.exit(main())
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/main.py", line 125, in main
    rv = omero.cli.argv()
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/cli.py", line 1784, in argv
    cli.invoke(args[1:])
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/cli.py", line 1222, in invoke
    stop = self.onecmd(line, previous_args)
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/cli.py", line 1299, in onecmd
    self.execute(line, previous_args)
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/cli.py", line 1381, in execute
    args.func(args)
  File "/data/idr0001/omero_cli_zarr/omero-cli-zarr/src/omero_zarr/cli.py", line 125, in _wrapper
    return func(self, *args, **kwargs)
  File "/data/idr0001/omero_cli_zarr/omero-cli-zarr/src/omero_zarr/cli.py", line 333, in polygons
    plate_shapes_to_zarr(plate, ["Polygon"], args)
  File "/data/idr0001/omero_cli_zarr/omero-cli-zarr/src/omero_zarr/masks.py", line 113, in plate_shapes_to_zarr
    saver.save(list(masks.values()), args.label_name)
  File "/data/idr0001/omero_cli_zarr/omero-cli-zarr/src/omero_zarr/masks.py", line 334, in save
    labels, fill_colors, properties = self.masks_to_labels(
  File "/data/idr0001/omero_cli_zarr/omero-cli-zarr/src/omero_zarr/masks.py", line 565, in masks_to_labels
    raise Exception(
Exception: Shape 624314 overlaps with existing labels

[will-moore]

Run mkngff on idr0125-pilot with

$ sudo mkdir /idr0001 && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr0001 /idr0001
$ ls /idr0001/zarr/
JL_120731_S6A.ome.zarr  JL_120731_S6B.ome.zarr

as omero-server

omero mkngff sql 16452 --clientpath="https://uk1s3.embassy.ebi.ac.uk/idr0001/zarr/JL_120731_S6B.ome.zarr" "/idr0001/zarr/JL_120731_S6B.ome.zarr" > "idr0001/16452.sql"

$ psql -U omero -d idr -h $DBHOST -f idr0001/16452.sql 
UPDATE 576
BEGIN
 mkngff_fileset 
----------------
        5289241
(1 row)

COMMIT


$ omero mkngff symlink /data/OMERO/ManagedRepository 16452 "/idr0001/zarr/JL_120731_S6B.ome.zarr" --bfoptions
...
Creating symlink /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-19.109_mkngff/JL_120731_S6B.ome.zarr -> /idr0001/zarr/JL_120731_S6B.ome.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-19.109
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-19.109_mkngff/JL_120731_S6B.ome.zarr.bfoptions

http://localhost:1040/webclient/?show=image-1230224 ...

Fails with:

2024-01-10 09:00:57,168 INFO  [                      omero.cmd.SessionI] (.Server-11) Unregistered servant:078c4a67-66a5-46f8-822f-8e196ea78f82/a88d46c1-8860-4398-b072-50d9939ab731omero.api.IQuery(omero.api._IQueryTie@cbaf0b18)
2024-01-10 09:00:57,168 INFO  [                      omero.cmd.SessionI] (.Server-11) Removed servant from adapter: a88d46c1-8860-4398-b072-50d9939ab731omero.api.IQuery
2024-01-10 09:01:05,954 DEBUG [                   loci.formats.Memoizer] (l.Server-0) start[1704877049382] time[216572] tag[loci.formats.Memoizer.setId]
2024-01-10 09:01:05,954 DEBUG [                   loci.formats.Memoizer] (.Server-10) start[1704876989274] time[276679] tag[loci.formats.Memoizer.setId]
2024-01-10 09:01:05,956 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-0) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-19.109_mkngff/JL_120731_S6B.ome.zarr/.zattrs
2024-01-10 09:01:05,956 ERROR [         ome.io.bioformats.BfPixelBuffer] (.Server-10) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-19.109_mkngff/JL_120731_S6B.ome.zarr/.zattrs
2024-01-10 09:01:05,957 ERROR [                ome.io.nio.PixelsService] (.Server-10) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-19.109_mkngff/JL_120731_S6B.ome.zarr/.zattrs
java.lang.RuntimeException: java.lang.ClassCastException: class java.lang.Long cannot be cast to class java.lang.String (java.lang.Long and java.lang.String are in module java.base of loader 'bootstrap')
        at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
        at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
        at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
        at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)
        at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571)
        at ome.services.RenderingBean$12.doWork(RenderingBean.java:2205)
        at jdk.internal.reflect.GeneratedMethodAccessor295.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)

[will-moore]

Error is raised at https://github.com/ome/omero-romio/blob/1d30fafc4e06c5511cfbb24c25a753925ffb2eb4/src/main/java/ome/io/bioformats/BfPixelBuffer.java#L79
but that's not where the ClassCastException comes from.

[sbesson]

Error is raised at https://github.com/ome/omero-romio/blob/1d30fafc4e06c5511cfbb24c25a753925ffb2eb4/src/main/java/ome/io/bioformats/BfPixelBuffer.java#L79
but that's not where the ClassCastException comes from.

I suspect the error comes from the underlying reader and is rethrown from the Bio-Formats pixel buffer. Is there more information in the following lines of the stack trace?

[will-moore]

Ah, yes...

Caused by: java.lang.ClassCastException: class java.lang.Long cannot be cast to class java.lang.String (java.lang.Long and java.lang.String are in module java.base of loader 'bootstrap')
        at loci.formats.in.ZarrReader.parsePlate(ZarrReader.java:755)
        at loci.formats.in.ZarrReader.initFile(ZarrReader.java:353)
        at loci.formats.FormatReader.setId(FormatReader.java:1443)
        at loci.formats.ImageReader.setId(ImageReader.java:849)
        at ome.io.nio.PixelsService$3.setId(PixelsService.java:869)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
        at loci.formats.ChannelFiller.setId(ChannelFiller.java:234)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
        at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:293)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
        at loci.formats.Memoizer.setId(Memoizer.java:690)
        at ome.io.bioformats.BfPixelsWrapper.<init>(BfPixelsWrapper.java:52)
        at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:73)
        ... 82 common frames omitted

[will-moore]

That line is https://github.com/ome/ZarrReader/blob/main/src/loci/formats/in/ZarrReader.java#L755

String acqStartTime = (String) acquistion.get("starttime");

The schema states that this is an integer:
https://github.com/ome/ngff/blob/main/0.4/schemas/plate.schema#L35

cc @dgault

[will-moore]

We want to create a copy of the original data without ...2.flex files but without duplicating all the data!

On pilot-zarr1-dev

export PLATE=JL_120801_S7A
cd /data/idr0001/raw

mkdir $PLATE
cd $PLATE
for i in $(ls /uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/); do
  mkdir "$i"
  for f in $(ls "/uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/$i"); do
    ln -s "/uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/$i/$f" "$i/$f"
  done
  rm "$i"/*2.flex
done

$ ~/bioformats2raw-0.6.0-24/bin/bioformats2raw "JL_120801_S7A/Meas_01(2012-08-01_10-57-32)/001001001.flex"  ngff/JL_120801_S7A.ome.zarr
...

Conversion took approx 30 mins.

(bioformats2raw) [wmoore@pilot-zarr1-dev ngff]$ ~/mc cp -r /data/idr0001/raw/ngff/JL_120801_S7A.ome.zarr uk1s3/idr0001/zarr
...7A.ome.zarr/OME/METADATA.ome.xml: 47.15 GiB / 47.15 GiB ━━━━━━━━━━━━━━━━ 49.82 MiB/s 16m9s