-
Notifications
You must be signed in to change notification settings - Fork 2
Complete a deposit stuck in the "depositing state"
If you don't see any related errors or alerts in Honeybadger (for e.g. H2 or sdr-api), the H2 Sidekiq dashboard, or the logs, the following may apply:
If a work gets stuck in the "depositing" state, but checking in Argo confirms the object is actually done with accessioning, and the problem is just that H2 has likely missed this (due to a dropped RabbitMQ message or some other problem), you can force H2 to set this object to the deposited state. To manually set the object in H2 to a completed state, starting with the work id, head to the H2 Rails console:
work_id = 123
object = Work.find(work_id)
DepositCompleter.complete(object_version: object.head)
This is what happens in the DepositCompleteJob which is what is triggered when the RabbitMQ message is received and processed.
When you see a DepositJob failure on a metadata-only update that is trying to find files for accessining
NOTE: we think we may have remediated the bad data that caused this error, when we ran across it in May 2025. And the buggy code or messaging issues that introduced the bad data may be fixed. It looked like all of the work versions we discovered when we first ran across the issue were created in 2023, when there was quite a bit of flux in the H2 codebase (both Globus related and otherwise). But if it comes up again...
If you see an error when e.g. DepositJob tries to update metadata in sdr-api using sdr-client (the error might be a fairly generic 502 bad gateway or other 500), and a corresponding error in sdr-api from the ResourcesController about an unavailable Globus path, the following may apply:
Example H2 Honeybadger alert for this case: https://app.honeybadger.io/projects/77112/faults/120537619
RuntimeError: unexpected response: 502 {"status":"error", "message": "Bad Gateway"}
stemming from from https://github.com/sul-dlss/happy-heron/blob/049d2b914327587969e5dce31dbd03bdf631335a/app/jobs/deposit_job.rb#L68
Example sdr-api HB alert for this case: https://app.honeybadger.io/projects/67994/faults/120537617
ResourcesController::GlobusNotFoundError: Globus file [/globus/sdr_ingest/prod/uploads//DTU_dropbox/DTU_1978.zip] not found.
If this is the case, it's possible that a deposit completed message was dropped, as in the first case in this wiki entry, but that the remediation was just an update of the WorkVersion's state to deposited, without having run DepositComplete.complete. This can potentially leave the file attachments in a state that is inconsistent with expectations: once a version is deposited, we expect the service_name for its attached files' ActiveStorage::Blobs to be preservation. If it instead e.g. globus or local, this can cause sdr-api and H2 to attempt to (re-)ingest files it does not need to ingest for the attempted versioning action.
You can query to see if the druid in question may be in this particular bad state:
WorkVersion.joins(:work, attached_files: [:file_blob]).where(state: "deposited").where.not('active_storage_blobs.service_name': 'preservation').pluck(:druid, :title, 'work_versions.id', :version, :service_name)If this gets you any results, they should be along the lines of:
[["druid:ps239gf1021", "How Credentialing Pathways Exacerbate California’s Teacher Shortage", 8916, 1, "local"],
["druid:wf145tx1670", "Morphological Analysis of Axo-Axonic Cell Variability", 8452, 1, "local"],
["druid:xf213fn4785", "Identification of DNA Termini in Sequencing Data through Combined Analysis of End Capture and Local Strand Bias", 8451, 1, "local"],
["druid:st754nd0738", "Guide to Wittenberg University Catalogs, etc.", 8835, 1, "local"],
["druid:zj843xz2372", "Stanford Environmental Research 2019 Year in Review", 8837, 1, "local"],
["druid:gj714cj0414", "Dataset Accompanying \"Platonic Distance:Intrinsic Object-Centric Image Similarity\"", 9235, 3, "globus"],
["druid:yc915nm6757", "Adapting SnapDx Platform For Low-Cost, Electricity-Free Molecular Amplification Based Detection of Schistosomiasis in LMIC Settings", 8977, 1, "local"],
["druid:rr555jy1718", "Guide to University of New Mexico Catalogs, etc.", 8297, 1, "local"]]If your druid is in that list, then for the work version in question, you can try this call to fix the various AttachedFiles' Blob service_name values for the WorkVersion:
work_version_id = '' # a `WorkVersion#id` that appears in the list above that is also associated with the `Work` for your failing `DepositJob`
WorkVersion.find(work_version_id).switch_to_preserved_items!- https://stanfordlib.slack.com/archives/C69UZCJ8M/p1747835220997999 (where problem was originally raised)
- https://stanfordlib.slack.com/archives/C09M7P91R/p1748373189199159
- https://stanfordlib.slack.com/archives/C09M7P91R/p1748040965356859
- https://stanfordlib.slack.com/archives/C69UZCJ8M/p1748014549261959 (where @andrewjbtw nudged us toward suspicion of deposit-related state changes)