some zip files failed size check when nightly audit triggered backfill to new GCP endpoint. but zips seem to have been created correctly?

[honeybadger]

@jmartin-sul is happy to pair on this! solution may be similar to #2437. one thing that's different from that ticket: i don't think these jobs ended up in the retry queue, so i don't have a druid list handy. but i believe we can query for these by looking for zipped_moab_versions rows that have no child zip_parts rows?

investigating, will likely remediate the week of june 2, see https://stanfordlib.slack.com/archives/C09M7P91R/p1748621014733859

TLDR for now: the zip files seem to be there, and from a couple spot checks, appear to be complete. i suspect the issue is some sort of file system lag causing occasional problems for the zip file size sanity check that's run after the zip is made.

original title from recent HB alert:

[preservation_catalog/prod] RuntimeError: zip size (0) is smaller than the moab version size (38605)! zipmaker failure   adding: mm814tx0636/v0004/ (stored 0%)  adding: mm814tx0636/v0004/manifests/ (stored 0%)  adding: mm814tx0636/v0004/manifests/versionAdditions.xml (stored 0%)  adding: mm814tx0636/v0004/manifests/signatureCatalog.xml (stored 0%)  adding: mm814tx0636/v0004/manifests/manifestInventory.xml (stored 0%)  adding: mm814tx0636/v0004/manifests/fileInventoryDifference.xml (stored 0%)  addi...

Backtrace

[PROJECT_ROOT]/app/models/replication/druid_version_zip.rb:59 :in `Replication::DruidVersionZip#create_zip!`
[PROJECT_ROOT]/app/models/replication/druid_version_zip.rb:43 :in `Replication::DruidVersionZip#find_or_create_zip!`
[PROJECT_ROOT]/app/jobs/replication/zipmaker_job.rb:11 :in `Replication::ZipmakerJob#find_or_create_zip!`
[PROJECT_ROOT]/app/jobs/replication/zipmaker_job.rb:35 :in `Replication::ZipmakerJob#perform` 

View full backtrace and more info at honeybadger.io

other alerts for pretty much the same error:

• https://app.honeybadger.io/projects/54415/faults/121092715 -- "Errno::ENOSPC: No space left on device @ rb_sysopen - /sdr-transfers/xt/299/pt/7593/xt299pt7593.v0003.zip.md5"
• https://app.honeybadger.io/projects/54415/faults/121092422 -- "Errno::ENOSPC: No space left on device @ dir_s_mkdir - /sdr-transfers/qs/086"
• https://app.honeybadger.io/projects/54415/faults/121092421 -- "RuntimeError: zipmaker failure adding: nb589cb1038/v0001/ (stored 0%) ..."