Bio-Formats cache regeneration walkthrough

[will-moore]

Following https://github.com/IDR/deployment/blob/master/docs/operating-procedures.md#bio-formats-cache-regeneration with step by step notes...

The instructions there distribute the work across all 5 proxy servers..

Followed instructions exactly as written on idr-testing until:

for i in $(cat nodes); do ssh $i 'pip install --user omero-cli-render'; done
bash: pip: command not found
bash: pip: command not found

We can probably skip this step if we use the one shared python environment at /opt/omero/server/venv3 for subsequent steps...

[wmoore@test120-omeroreadwrite ~]$ sudo rm -rf /data/BioFormatsCache/*

Checking we have same jars on all servers... e.g. ZarrReader...

for i in $(cat nodes); do ssh $i 'md5sum /opt/omero/server/OMERO.server/lib/server/OMEZarr*'; done
ce6e7e67626f8c5cc59085b0282ba9cc  /opt/omero/server/OMERO.server/lib/server/OMEZarrReader.jar
ce6e7e67626f8c5cc59085b0282ba9cc  /opt/omero/server/OMERO.server/lib/server/OMEZarrReader.jar
ce6e7e67626f8c5cc59085b0282ba9cc  /opt/omero/server/OMERO.server/lib/server/OMEZarrReader.jar
ce6e7e67626f8c5cc59085b0282ba9cc  /opt/omero/server/OMERO.server/lib/server/OMEZarrReader.jar
ce6e7e67626f8c5cc59085b0282ba9cc  /opt/omero/server/OMERO.server/lib/server/OMEZarrReader.jar

Deletion has been running for an hour and a half now:

[wmoore@test120-omeroreadwrite ~]$ sudo rm -rf /data/BioFormatsCache/*

[will-moore]

As suggested, I tried instead to move data initially, then delete later.
Left this running in a screen...
But it seems this was actually copying data from one mounted location to another?

[wmoore@test120-omeroreadwrite ~]$ sudo mkdir /BFCacheToDelete2
[wmoore@test120-omeroreadwrite ~]$ sudo mv /data/BioFormatsCache/data/OMERO/ManagedRepository/demo_2 /BFCacheToDelete2
...
mv: cannot create directory ‘/BFCacheToDelete2/demo_2/Blitz-0-Ice.ThreadPool.Server-165’: No space left on device
mv: cannot create directory ‘/BFCacheToDelete2/demo_2/Blitz-0-Ice.ThreadPool.Server-199’: No space left on device
mv: cannot create directory ‘/BFCacheToDelete2/demo_2/Blitz-0-Ice.ThreadPool.Server-179’: No space left on device
mv: cannot create directory ‘/BFCacheToDelete2/demo_2/Blitz-0-Ice.ThreadPool.Server-202’: No space left on device
mv: cannot create directory ‘/BFCacheToDelete2/demo_2/Blitz-0-Ice.ThreadPool.Server-180’: No space left on device
mv: cannot create directory ‘/BFCacheToDelete2/demo_2/Blitz-0-Ice.ThreadPool.Server-200’: No space left on device
mv: cannot create directory ‘/BFCacheToDelete2/demo_2/Blitz-0-Ice.ThreadPool.Server-206’: No space left on device

Restarted $ sudo rm -rf /data/BioFormatsCache/* in a screen...

Meanwhile cleanup, $ sudo rm -rf /BFCacheToDelete2

[will-moore]

Seems that parallel wasn't installed on idr-testing...

[wmoore@test120-proxy ~]$ which parallel
/usr/bin/which: no parallel in (/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/wmoore/.local/bin:/home/wmoore/bin)

[wmoore@test120-proxy ~]$ sudo yum install https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/p/parallel-20160222-1.el7.noarch.rpm
...
[wmoore@test120-proxy ~]$ which parallel
/usr/local/bin/parallel

Try again..

screen -dmS cache parallel --eta --sshloginfile nodes -a ids.txt --results /tmp/cache/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force'

screen -r

Computers / CPU threads / Max jobs to run
1:omeroreadonly-1 / 1 / 10
2:omeroreadonly-2 / 1 / 10
3:omeroreadonly-3 / 1 / 10
4:omeroreadonly-4 / 1 / 10
5:omeroreadwrite / 1 / 10

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 21698s Left: 21691 AVG: 2.88s  omeroreadonly-1:10/2/20%/29.0s  omeroreadonly-2:10/5/25%/11.6s  omeroreadonly-3:10/0/17%/0.0s  omeroreadonly-4:10/1/18%/58.0s  omeroreadwrite:10/0/17%/0.0s

[will-moore]

Seems pretty unhappy... Lots of DatabaseBusyException...

[wmoore@test120-proxy ~]$ grep Error /tmp/cache/**/**/* | wc
     91     712   14456

[wmoore@test120-proxy ~]$ grep Error /tmp/cache/**/**/*
...
/tmp/cache/1/Image:608472/stderr:FileNotFoundError: [Errno 2] No such file or directory: path('/home/wmoore/omero/sessions/localhost/public/5e2e34a1-4b10-49e8-a07d-132d164df892')
/tmp/cache/1/Image:608856/stderr:FileNotFoundError: [Errno 2] No such file or directory: path('/home/wmoore/omero/sessions/localhost/public/5e2e34a1-4b10-49e8-a07d-132d164df892')
/tmp/cache/1/Image:610008/stderr:FileNotFoundError: [Errno 2] No such file or directory: path('/home/wmoore/omero/sessions/localhost/public/5e2e34a1-4b10-49e8-a07d-132d164df892')
/tmp/cache/1/Image:63089/stderr:FileExistsError: [Errno 17] File exists: path('/home/wmoore/omero/tmp')
/tmp/cache/1/Image:63089/stderr:FileExistsError: [Errno 17] File exists: path('/tmp/omero/tmp')
/tmp/cache/1/Image:63089/stderr:Error loading: /opt/omero/server/venv3/lib64/python3.6/site-packages/omero/plugins/_metadata_deprecated.py
/tmp/cache/1/Image:667768/stderr:FileNotFoundError: [Errno 2] No such file or directory: path('/home/wmoore/omero/sessions/localhost/public/05ee3d02-7fd0-4ec7-ad69-70692dc7bb7e')
/tmp/cache/1/Image:68465/stderr:FileNotFoundError: [Errno 2] No such file or directory: path('/home/wmoore/omero/sessions/localhost/public/577d172f-af4a-43b6-a5af-2f2ce42beef1')
/tmp/cache/1/Image:69233/stderr:FileNotFoundError: [Errno 2] No such file or directory: path('/home/wmoore/omero/sessions/localhost/public/577d172f-af4a-43b6-a5af-2f2ce42beef1')
/tmp/cache/1/Image:693144/stderr:FileNotFoundError: [Errno 2] No such file or directory: path('/home/wmoore/omero/sessions/localhost/public/97488570-6c46-4eec-8874-12ee239091ff')
...

[wmoore@test120-proxy ~]$ cat /tmp/cache/1/Image:98291/stderrr
cat: /tmp/cache/1/Image:98291/stderrr: No such file or directory
[wmoore@test120-proxy ~]$ cat /tmp/cache/1/Image:98291/stderr
Traceback (most recent call last):
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/plugins/sessions.py", line 622, in attach
    rv = store.attach(server, name, uuid, set_current=set_current)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/util/sessions.py", line 349, in attach
    set_current=set_current)
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero/util/sessions.py", line 385, in create
    ec = sf.getAdminService().getEventContext()
  File "/opt/omero/server/venv3/lib64/python3.6/site-packages/omero_api_IAdmin_ice.py", line 2655, in getEventContext
    return _M_omero.api.IAdmin._op_getEventContext.invoke(self, ((), _ctx))
omero.DatabaseBusyException: exception ::omero::DatabaseBusyException
{
    serverStackTrace = ome.conditions.DatabaseBusyException: cannot create transaction


[wmoore@test120-proxy ~]$  grep DatabaseBusyException /tmp/cache/**/**/* | wc
    907   10422  135925

[will-moore]

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 46524s Left: 20435 AVG: 2.30s  omeroreadonly-1:10/284/22%/10.4s  omeroreadonly-2:10/146/11%/20.2s  omeroreadonly-3:10/63/5%/46.7s  omeroreadonly-4:10/141/11%/20.9s  omeroreadwrite:10/630/48%/4.7s 

[will-moore]

Cancelled:

[wmoore@test120-proxy ~]$ screen -X -S 2497.cache quit

Restarting...

for server in omeroreadwrite omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do ssh $server "sudo service omero-server restart"; done

[will-moore]

Re-running...

[wmoore@test120-proxy ~]$ rm -rf /tmp/cache/
[wmoore@test120-proxy ~]$ screen -dmS cache parallel --eta --sshloginfile nodes -a ids.txt --results /tmp/cache/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force
'

[will-moore]

[wmoore@test120-proxy ~]$ screen -r 3951.cache
...
Computers / CPU threads / Max jobs to run
1:omeroreadonly-1 / 1 / 10
2:omeroreadonly-2 / 1 / 10
3:omeroreadonly-3 / 1 / 10
4:omeroreadonly-4 / 1 / 10
5:omeroreadwrite / 1 / 10

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 13701s Left: 16084 AVG: 0.88s  omeroreadonly-1:10/458/8%/10.8s  omeroreadonly-2:10/401/7%/12.4s  omeroreadonly-3:10/322/5%/15.4s  omeroreadonly-4:10/440/7%/11.3s  omeroreadwrite:10/3994/70%/1.2s 

[will-moore]

[wmoore@test120-proxy ~]$ cat /tmp/cache/1/Image:10842059/stderr
usage: /opt/omero/server/venv3/bin/omero render test [-h] [--force] [--thumb]
                                                     object
/opt/omero/server/venv3/bin/omero render test: error: too few arguments
/tmp/par-job-5089_BCnlw: line 3: Image:10842059: command not found

Ooops - added newline in command above. Cancel and start again...

[wmoore@test120-proxy ~]$ rm -rf /tmp/cache/
[wmoore@test120-proxy ~]$ screen -dmS cache parallel --eta --sshloginfile nodes -a ids.txt --results /tmp/cache/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force'

[will-moore]

Looks like running OK now...

[wmoore@test120-proxy ~]$ grep ok /tmp/cache/**/**/*
...
/tmp/cache/1/Image:62321/stdout:ok: 62321 70.78979206085205 
/tmp/cache/1/Image:63089/stdout:ok: 63089 202.19777274131775 
/tmp/cache/1/Image:63857/stdout:ok: 63857 2.0913946628570557 
/tmp/cache/1/Image:64625/stdout:ok: 64625 1.5413122177124023 
/tmp/cache/1/Image:65393/stdout:ok: 65393 1.3958213329315186 

[wmoore@test120-proxy ~]$ grep ok /tmp/cache/**/**/* | wc
     50     150    3077

[wmoore@test120-proxy ~]$ grep Error /tmp/cache/**/**/*
/tmp/cache/1/Image:38517/stdout:fail: 38517 14.241586923599243 exception ::omero::ResourceError
/tmp/cache/1/Image:40053/stdout:fail: 40053 13.972737312316895 exception ::omero::ResourceError

[will-moore]

As discussed in IDR meeting now, decided to stop memo generation now and test with ~2000 filesets that have been processed so far.

[wmoore@test120-proxy ~]$ screen -X -S 14861.cache quit

[will-moore]

Exception on omeroreadonly-4

2024-02-15 09:53:44,486 INFO  [                 org.perf4j.TimingLogger] (.Server-31) start[1707990819611] time[4874] tag[omero.call.success.ome.logic.QueryImpl.findAllByQuery]
2024-02-15 09:53:44,486 INFO  [        ome.services.util.ServiceHandler] (.Server-31)  Rslt:    (ome.model.core.Image:Id_10496304)
2024-02-15 09:53:44,486 INFO  [    ome.security.basic.BasicEventContext] (.Server-26)  cctx:    group=-1
2024-02-15 09:53:44,492 INFO  [        ome.services.util.ServiceHandler] (.Server-38)  Meth:    interface ome.api.IQuery.findAllByQuery
2024-02-15 09:53:44,492 INFO  [        ome.services.util.ServiceHandler] (.Server-38)  Args:    [select i from Image as i left outer join i.wellSamples as ws join ws.well as well join well.plate as, PARAMS:F(o0l1):tag_text=Study Example Image id
=2501 ]
2024-02-15 09:53:44,495 INFO  [         ome.security.basic.EventHandler] (.Server-26)  Auth:    user=52,group=-1,event=null(User),sess=b3bff02f-85c2-4591-bbf0-5b73d2f11367
2024-02-15 09:53:44,647 INFO  [                 org.perf4j.TimingLogger] (.Server-36) start[1707990819845] time[4801] tag[omero.call.exception]
2024-02-15 09:53:44,647 WARN  [        ome.services.util.ServiceHandler] (.Server-36) Unknown exception thrown.

org.springframework.dao.DataAccessResourceFailureException: Hibernate operation: could not execute query; SQL [select image0_.id as id60_, image0_.acquisitionDate as acquisit2_60_, image0_.archived as archived60_, image0_.description as descript
4_60_, image0_.creation_id as creation10_60_, image0_.external_id as external11_60_, image0_.group_id as group12_60_, image0_.owner_id as owner13_60_, image0_.permissions as permissi5_60_, image0_.update_id as update14_60_, image0_.experiment as
 experiment60_, image0_.fileset as fileset60_, image0_.format as format60_, image0_.imagingEnvironment as imaging18_60_, image0_.instrument as instrument60_, image0_.name as name60_, image0_.objectiveSettings as objecti20_60_, image0_.partial as
 partial60_, image0_.series as series60_, image0_.stageLabel as stageLabel60_, image0_.version as version60_ from image image0_ left outer join wellsample wellsample1_ on image0_.id=wellsample1_.image and 
( 
  1 = ? OR 
  1 = ? OR 
  (wellsample1_.group_id in (?)) OR 
  (wellsample1_.owner_id = ? AND (select (__g.permissions & 1024) = 1024 from experimentergroup __g where __g.id = wellsample1_.group_id)) OR 
  (wellsample1_.group_id in (?,?) AND (select (__g.permissions & 64) = 64 from experimentergroup __g where __g.id = wellsample1_.group_id)) OR 
  ((select (__g.permissions & 4) = 4 from experimentergroup __g where __g.id = wellsample1_.group_id)) 
) inner join well well2_ on wellsample1_.well=well2_.id inner join plate plate3_ on well2_.plate=plate3_.id left outer join screenplatelink screenlink4_ on plate3_.id=screenlink4_.child and 
( 
  1 = ? OR 
  1 = ? OR 
  (screenlink4_.group_id in (?)) OR 
  (screenlink4_.owner_id = ? AND (select (__g.permissions & 1024) = 1024 from experimentergroup __g where __g.id = screenlink4_.group_id)) OR 
  (screenlink4_.group_id in (?,?) AND (select (__g.permissions & 64) = 64 from experimentergroup __g where __g.id = screenlink4_.group_id)) OR 
  ((select (__g.permissions & 4) = 4 from experimentergroup __g where __g.id = screenlink4_.group_id)) 
) inner join screen screen5_ on screenlink4_.parent=screen5_.id left outer join imageannotationlink annotation6_ on image0_.id=annotation6_.parent and 
( 
  1 = ? OR 
  1 = ? OR 
  (annotation6_.group_id in (?)) OR 
  (annotation6_.owner_id = ? AND (select (__g.permissions & 1024) = 1024 from experimentergroup __g where __g.id = annotation6_.group_id)) OR 
  (annotation6_.group_id in (?,?) AND (select (__g.permissions & 64) = 64 from experimentergroup __g where __g.id = annotation6_.group_id)) OR 
  ((select (__g.permissions & 4) = 4 from experimentergroup __g where __g.id = annotation6_.group_id)) 
) inner join annotation annotation7_ on annotation6_.child=annotation7_.id where 
( 
  1 = ? OR 
  1 = ? OR 
  (image0_.group_id in (?)) OR 
  (image0_.owner_id = ? AND (select (__g.permissions & 1024) = 1024 from experimentergroup __g where __g.id = image0_.group_id)) OR 
  (image0_.group_id in (?,?) AND (select (__g.permissions & 64) = 64 from experimentergroup __g where __g.id = image0_.group_id)) OR 
:
  ((select (__g.permissions & 4) = 4 from experimentergroup __g where __g.id = image0_.group_id)) 
) and screen5_.id=? and annotation7_.textValue=? order by well2_."column", well2_."row"]; ERROR: remaining connection slots are reserved for non-replication superuser connections
  Where: parallel worker; nested exception is org.postgresql.util.PSQLException: ERROR: remaining connection slots are reserved for non-replication superuser connections
  Where: parallel worker
        at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:251)
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73)
        at org.springframework.orm.hibernate3.HibernateAccessor.convertJdbcAccessException(HibernateAccessor.java:428)
        at org.springframework.orm.hibernate3.HibernateAccessor.convertHibernateAccessException(HibernateAccessor.java:414)
        at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:416)
        at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:342)
        at ome.logic.QueryImpl.execute(QueryImpl.java:176)
        at ome.logic.QueryImpl.findAllByQuery(QueryImpl.java:419)
        at jdk.internal.reflect.GeneratedMethodAccessor597.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:119)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:249)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:121)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        at com.sun.proxy.$Proxy111.findAllByQuery(Unknown Source)
        at jdk.internal.reflect.GeneratedMethodAccessor597.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:93)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        at com.sun.proxy.$Proxy111.findAllByQuery(Unknown Source)
        at jdk.internal.reflect.GeneratedMethodAccessor729.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:172)
:
        at ome.services.throttling.Callback.run(Callback.java:56)
        at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
        at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:140)
        at ome.services.blitz.impl.QueryI.findAllByQuery_async(QueryI.java:64)
        at jdk.internal.reflect.GeneratedMethodAccessor728.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at omero.cmd.CallContext.invoke(CallContext.java:85)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        at com.sun.proxy.$Proxy112.findAllByQuery_async(Unknown Source)
        at omero.api._IQueryTie.findAllByQuery_async(_IQueryTie.java:90)
        at omero.api._IQueryDisp.___findAllByQuery(_IQueryDisp.java:594)
        at omero.api._IQueryDisp.__dispatch(_IQueryDisp.java:733)
        at IceInternal.Incoming.invoke(Incoming.java:221)
        at Ice.ConnectionI.invokeAll(ConnectionI.java:2536)
        at Ice.ConnectionI.dispatch(ConnectionI.java:1145)
        at Ice.ConnectionI.message(ConnectionI.java:1056)
        at IceInternal.ThreadPool.run(ThreadPool.java:395)
        at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
        at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.postgresql.util.PSQLException: ERROR: remaining connection slots are reserved for non-replication superuser connections
  Where: parallel worker
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2422)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2167)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
        at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
        at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:155)
        at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:118)
        at bitronix.tm.resource.jdbc.proxy.PreparedStatementJavaProxy.executeQuery(PreparedStatementJavaProxy.java:102)
        at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
        at org.hibernate.loader.Loader.getResultSet(Loader.java:1953)
        at org.hibernate.loader.Loader.doQuery(Loader.java:802)
        at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
        at org.hibernate.loader.Loader.doList(Loader.java:2542)
        at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2276)
        at org.hibernate.loader.Loader.list(Loader.java:2271)
        at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:459)
        at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:365)
        at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
        at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1268)
        at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
        at ome.services.query.Query.doInHibernate(Query.java:253)
:
        at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:411)
        ... 64 common frames omitted
2024-02-15 09:53:44,648 INFO  [        ome.services.util.ServiceHandler] (.Server-36)  Excp:    org.springframework.dao.DataAccessResourceFailureException: Hibernate operation: could not execute query; SQL [select image0_.id as id60_, image0_.acquisitionDate as acquisit2_60_, image0_.archived as archived60_, image0_.description as descript4_60_, image0_.creation_id as creation10_60_, image0_.external_id as external11_60_, image0_.group_id as group12_60_, image0_.owner_id as owner13_60_, image0_.permissions as permissi5_60_, image0_.update_id as update14_60_, image0_.experiment as experiment60_, image0_.fileset as fileset60_, image0_.format as format60_, image0_.imagingEnvironment as imaging18_60_, image0_.instrument as instrument60_, image0_.name as name60_, image0_.objectiveSettings as objecti20_60_, image0_.partial as partial60_, image0_.series as series60_, image0_.stageLabel as stageLabel60_, image0_.version as version60_ from image image0_ left outer join wellsample wellsample1_ on image0_.id=wellsample1_.image and 
( 
  1 = ? OR 
  1 = ? OR 
  (wellsample1_.group_id in (?)) OR 
  (wellsample1_.owner_id = ? AND (select (__g.permissions & 1024) = 1024 from experimentergroup __g where __g.id = wellsample1_.group_id)) OR 
  (wellsample1_.group_id in (?,?) AND (select (__g.permissions & 64) = 64 from experimentergroup __g where __g.id = wellsample1_.group_id)) OR 
  ((select (__g.permissions & 4) = 4 from experimentergroup __g where __g.id = wellsample1_.group_id)) 
) inner join well well2_ on wellsample1_.well=well2_.id inner join plate plate3_ on well2_.plate=plate3_.id left outer join screenplatelink screenlink4_ on plate3_.id=screenlink4_.child and 
( 
  1 = ? OR 
  1 = ? OR 
  (screenlink4_.group_id in (?)) OR 
  (screenlink4_.owner_id = ? AND (select (__g.permissions & 1024) = 1024 from experimentergroup __g where __g.id = screenlink4_.group_id)) OR 
  (screenlink4_.group_id in (?,?) AND (select (__g.permissions & 64) = 64 from experimentergroup __g where __g.id = screenlink4_.group_id)) OR 
  ((select (__g.permissions & 4) = 4 from experimentergroup __g where __g.id = screenlink4_.group_id)) 
) inner join screen screen5_ on screenlink4_.parent=screen5_.id left outer join imageannotationlink annotation6_ on image0_.id=annotation6_.parent and 
( 
  1 = ? OR 
  1 = ? OR 
  (annotation6_.group_id in (?)) OR 
  (annotation6_.owner_id = ? AND (select (__g.permissions & 1024) = 1024 from experimentergroup __g where __g.id = annotation6_.group_id)) OR 
  (annotation6_.group_id in (?,?) AND (select (__g.permissions & 64) = 64 from experimentergroup __g where __g.id = annotation6_.group_id)) OR 
  ((select (__g.permissions & 4) = 4 from experimentergroup __g where __g.id = annotation6_.group_id)) 
) inner join annotation annotation7_ on annotation6_.child=annotation7_.id where 
( 
  1 = ? OR 
  1 = ? OR 
  (image0_.group_id in (?)) OR 
  (image0_.owner_id = ? AND (select (__g.permissions & 1024) = 1024 from experimentergroup __g where __g.id = image0_.group_id)) OR 
  (image0_.group_id in (?,?) AND (select (__g.permissions & 64) = 64 from experimentergroup __g where __g.id = image0_.group_id)) OR 
  ((select (__g.permissions & 4) = 4 from experimentergroup __g where __g.id = image0_.group_id)) 
) and screen5_.id=? and annotation7_.textValue=? order by well2_."column", well2_."row"]; ERROR: remaining connection slots are reserved for non-replication superuser connections
  Where: parallel worker; nested exception is org.postgresql.util.PSQLException: ERROR: remaining connection slots are reserved for non-replication superuser connections
  Where: parallel worker
2024-02-15 09:53:44,648 INFO  [    ome.security.basic.BasicEventContext] (.Server-33)  cctx:    group=-1
2024-02-15 09:53:44,657 INFO  [         ome.security.basic.EventHandler] (.Server-33)  Auth:    user=52,group=-1,event=null(User),sess=b3bff02f-85c2-4591-bbf0-5b73d2f11367
2024-02-15 09:53:44,659 INFO  [o.services.sessions.SessionContext$Count] (.Server-23) -Reference count: b3bff02f-85c2-4591-bbf0-5b73d2f11367=15

[will-moore]

Started again...

screen -dmS cache parallel --eta --sshloginfile nodes -a ids.txt --results /tmp/cache/ -j10 '/opt/omero/server/OMERO.server/bin/omero render -s localhost -u public -w public test --force'

Status after about 5 mins...

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 4107s Left: 20296 AVG: 0.20s  omeroreadonly-1:10/38/3%/8.3s  omeroreadonly-2:10/267/19%/1.2s  omeroreadonly-3:10/280/19%/1.1s  omeroreadonly-4:10/221/15%/1.4s  omeroreadwrite:10/597/41%/0.5s 

[will-moore]

16 hours later... apart from computer:1, percentages seem lower today than yesterday

Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 440639s Left: 19178 AVG: 23.45s  1:10/257/10%/230.1s  2:10/471/18%/125.6s  3:10/469/18%/126.1s  4:10/482/19%/122.7s  5:10/842/33%/70.2s