Testing mkngff on idr-testing

[will-moore]

Seb: "test120 now deployed with OMERO.server and available at https://idr-testing.openmicroscopy.org/".
As discussed at IDR meeting 11th September, good to test mkngff with ALL the NGFF filesets:

• idr0004 mkngff 2.5 hrs (46 x 100-well-Plate - 3mins per Plate) - setId 1 min per Fileset
• idr0010 - 148 x 384-well-Plates - mkngff 9 mins per plate 142/147 done
• idr0011 mkngff 3.5 hrs (182 x 48-well-Plate) 1 min per plate) - setId 1min per Plate. Improved to 40-50 secs
• idr0012 - 68 x 320-well-Plates (2 fields) - mkngff 6 mins per plate. Memo file took 42 minutes
• idr0013 - 538 x 384-well-Plates - mkngff 3 mins per plate x 538 => 26 hours. memo file took 11 minutes x 538 = 4 days
• idr0015 - 84 x 396-well-Plates
• idr0016 - 413 x 384-well-Plates (6 fields) mkngff 7.5 mins per Plate - x 413 => 51 hours
• idr0025 mkngff 3 x 96-well-Plates. setId 95 mins for a plate, 4 mins for Plate 2, 12 mins for Plate 3.
• idr0026 mkngff took 90 mins (111 images). setId took 26 secs for a single image
• idr0033 - mkngff took 18 hours for 12 x 384-well-plates (9 fields). memo file took 24 hours for a single Plate, then 12 hours for Plate 2 - Now down to 81 minutes
• idr0035 - mkngff took 6.5 hours (55 plates). setId took 1 hour for a single Plate
• idr0036 - 20 Plates
• idr0051 - 5 Images
• idr0054 - 3 Images
• idr0090 - 22 Plates
• idr0091 - 342 Images - on idr0125-pilot mkngff took few secs each. 28 mins total

View on https://idr-testing.openmicroscopy.org - Updated ZarrReader on 2023.09.25

• idr0004
• idr0011

[will-moore]

ssh idr-testing.openmicroscopy.org -At ssh omeroreadwrite -A

# also run this in another terminal:
ssh idr-testing.openmicroscopy.org -A -L 1080:omeroreadwrite:80

sudo mkdir /bia-integrator-data && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-data

Install mkngff latest PR IDR/omero-mkngff#8 into /opt/omero/server/venv3..

sudo -u root -s
source /opt/omero/server/venv3/bin/activate

pip install 'omero-mkngff @ git+https://github.com/will-moore/omero-mkngff@always_update_pixels'
... Resolved https://github.com/will-moore/omero-mkngff to commit 231431164e3692298864fcc52fb3f0c663c8f595
Successfully installed omero-mkngff-0.1.0.dev0

[will-moore]

Going to test idr0051 (3 images), following IDR/omero-mkngff#2 workflow...

idr0051.csv

idr0054/Tonsil_1.ome.zarr,S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39,1591301
idr0054/Tonsil_2.ome.zarr,S-BIAD800/e24d29ea-0e39-4368-a2e4-5e6320c91573,1591302
idr0054/Tonsil_3.ome.zarr,S-BIAD800/f49ada41-43bf-47ff-99b9-bdf8cc311ce3,5025553

$ psql -U omero -d idr -h $DBHOST -c "select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 1) x order by x.id asc limit 1;"

                 uuid
--------------------------------------
 01b61552-49fc-4543-bf40-...
(1 row)

$ psql -U omero -d idr -h $DBHOST -c "select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 3) x order by x.id asc limit 3;"
                 uuid                 
--------------------------------------
 9dee647a-d1ed-483b-8e67-...
 7b025064-588b-4d17-8eb4-...
 01b61552-49fc-4543-bf40-...
(3 rows)

for r in $(cat $IDRID.csv); do
>   biapath=$(echo $r | cut -d',' -f2)
>   uuid=$(echo $biapath | cut -d'/' -f2)
>   fsid=$(echo $r | cut -d',' -f3)
>   omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET $fsid "/bia-integrator-data/$biapath/$uuid.zarr" > "$fsid.sql"
> done
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15 // 15-27-40.023 for fileset 1591301
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr -> /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-10/2019-03/15 // 15-28-44.081 for fileset 1591302
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-10/2019-03/15/15-28-44.081
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-10/2019-03/15/15-28-44.081_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-10/2019-03/15/15-28-44.081_mkngff/e24d29ea-0e39-4368-a2e4-5e6320c91573.zarr -> /bia-integrator-data/S-BIAD800/e24d29ea-0e39-4368-a2e4-5e6320c91573/e24d29ea-0e39-4368-a2e4-5e6320c91573.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-13/2021-06/07 // 23-05-45.568 for fileset 5025553
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-13/2021-06/07/23-05-45.568
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-13/2021-06/07/23-05-45.568_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-13/2021-06/07/23-05-45.568_mkngff/f49ada41-43bf-47ff-99b9-bdf8cc311ce3.zarr -> /bia-integrator-data/S-BIAD800/f49ada41-43bf-47ff-99b9-bdf8cc311ce3/f49ada41-43bf-47ff-99b9-bdf8cc311ce3.zarr

UPDATE 1
BEGIN
 mkngff_fileset 
----------------
        6311582
(1 row)
COMMIT
UPDATE 1
BEGIN
 mkngff_fileset 
----------------
        6311583
(1 row)
COMMIT
UPDATE 1
BEGIN
 mkngff_fileset 
----------------
        6311584
(1 row)
COMMIT

[will-moore]

Image above is not viewable...
https://idr-testing.openmicroscopy.org/webclient/imgData/5025551/
returns

"Exception": "Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml"

[sbesson]

Was the bucket mounted both on the read-write VM and the read-only VMs (which are the ones accessed by the idr-testing endpoint)?

[will-moore]

Update ZarrReader to include all current PRs, as described... IDR/omero-mkngff#2 (comment)

wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/lastSuccessfulBuild/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar
...

[will-moore]

@sbesson no, I'll try... - I also wondered if the ZarrReader updates need to be applied across all VMs?

[sbesson]

@sbesson no, I'll try... - I also wondered if the ZarrReader updates need to be applied across all VMs?

If you are departing from the release 0.3.1 artifact was deployed as part of IDR/deployment#380, the same JAR will definitely need to be copied under lib/server on all VMs and the omero-server service restarted accordingly

[will-moore]

Mount bia-integrator-data on all the readonly servers...

ssh -A idr-testing.openmicroscopy.org

for server in omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do ssh $server "sudo mkdir /bia-integrator-data && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-data"; done

@sbesson What about the creation of symlinks by mkngff? Does that need to happen on multiple servers?

[sbesson]

That should work out of the box as the OMERO binary repository should be shared between the read-write VM and all read-only VMs.

[will-moore]

I updated ZarrReader on omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4 as at IDR/omero-mkngff#2 (comment):

wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/lastSuccessfulBuild/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar

But I'm still seeingError instantiating pixel buffer at https://idr-testing.openmicroscopy.org/webclient/imgData/5025551/

I'was trying to check the Blitz.log on omeroreadwrite at the same time as refreshing that URL, but I don't see anything because I guess this is being handled by a different server. I guess I have to try them in turn...

[will-moore]

On omeroreadonly-2 I saw this $ less /opt/omero/server/OMERO.server/var/log/Blitz-0.log...

2023-09-07 09:44:29,669 INFO  [         ome.security.basic.EventHandler] (l.Server-4)  Auth:    user=52,group=3,event=null(User),sess=bb36fb91-d187-4988-8d0e-ef595314e803
2023-09-07 09:44:29,670 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-4) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.
ome.xml
2023-09-07 09:44:30,259 INFO  [                loci.formats.ImageReader] (l.Server-4) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-09-07 09:44:33,844 WARN  [              loci.formats.FormatHandler] (l.Server-4) Mismatch with group key paths and original OME-XML metadata, original ordering wont be maintained
2023-09-07 09:44:34,734 DEBUG [                   loci.formats.Memoizer] (l.Server-4) start[1694079869675] time[5059] tag[loci.formats.Memoizer.setId]
2023-09-07 09:44:34,735 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-4) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/ME
TADATA.ome.xml
2023-09-07 09:44:34,735 ERROR [                ome.io.nio.PixelsService] (l.Server-4) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.om
e.xml
java.lang.RuntimeException: java.lang.ClassCastException: class com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy cannot be cast to class com.esotericsoftware.kryo.util.DefaultInstantiatorStrategy (com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy and com.eso
tericsoftware.kryo.util.DefaultInstantiatorStrategy are in unnamed module of loader 'app')
        at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
        at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
        at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
        at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)
        at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571)
        at ome.services.RenderingBean$12.doWork(RenderingBean.java:2205)
        at jdk.internal.reflect.GeneratedMethodAccessor224.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
...
	at IceInternal.ThreadPool.run(ThreadPool.java:395)
	at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassCastException: class com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy cannot be cast to class com.esotericsoftware.kryo.util.DefaultInstantiatorStrategy (com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy and com.esotericsoftware.kryo.util.DefaultInstantiatorStrategy are in unnamed module of loader 'app')
	at loci.formats.Memoizer$KryoDeser.<init>(Memoizer.java:125)
	at loci.formats.Memoizer.getDeser(Memoizer.java:789)
	at loci.formats.Memoizer.saveMemo(Memoizer.java:990)
	at loci.formats.Memoizer.setId(Memoizer.java:733)
	at ome.io.bioformats.BfPixelsWrapper.<init>(BfPixelsWrapper.java:52)
	at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:73)
	... 82 common frames omitted
2023-09-07 09:43:18,832 INFO  [                 org.perf4j.TimingLogger] (l.Server-0) start[1694079795148] time[3684] tag[omero.call.exception]

[will-moore]

David (slack) "Kryo will be related to the memoizer, I haven’t seen that particular issue though, perhaps a version mismatch with the dependency?
if it is also using the latest merge of Bio-Formats then that may be using different versions"

[sbesson]

One thing worth investigating is that the latest OMERO.server release, 5.6.8, was deployed on test120 as opposed to OMERO 5.6.7 on pilot-idr0125. In terms of underlying library changes, Bio-Formats was upgraded from 6.12.0 to 6.14.0 - see https://www.openmicroscopy.org/2023/07/25/omero-5-6-8.html which in turns included a large update of low-level dependencies and notably kryo - see https://www.openmicroscopy.org/2023/05/02/bio-formats-6-13-0.html.

Would it be worth to revert test120 to OMERO.server 5.6.7 to confirm whether the issue is related to this dependency upgrade ?