Complete remaining steps to decommission was-thumbnail

[andrewjbtw]

Decommissioning was-thumbnail was started in a proxy ticket in Argo sul-dlss/argo#2091

Some of this work has been done already. Using this ticket to make a checklist of what's needed to finish the decommissioning process:

• Replace was-thumbnail-based embed viewer with viewer that makes use of the one thumbnail created during seed accessioning [done]
• Stop running the wasSeedDisseminationWF, which calls was-thumbnail-service (see was_robot [README] (https://github.com/sul-dlss/was_robot_suite/blob/master/README.md#wasseeddissemination))
• Determine if any other workflow steps are connecting to was-thumbnail
• Check for any other was-thumbnail connections that might be made in the background (does SWAP expect some interaction with was-thumbnail?)
• Ask OPs to dissociate was-thumbnail VM from IP to see if there is any connections left that we are unaware of
• Run an accessioning test - if web archiving accessioning still works, we can remove the was-thumbnail VMs, deprecate this repo
• Clean up Stacks to remove the thumbnails left there by was-thumbnail

[jcoyne]

I think the second step can be accomplished by removing this clause:
https://github.com/sul-dlss/was_robot_suite/blob/master/robots/was_dissemination/start_special_dissemination.rb#L18

And the one step in that workflow: https://github.com/sul-dlss/was_robot_suite/blob/master/robots/was_seed_dissemination/update_thumbnail_generator.rb

[jmartin-sul]

the plan is for the first responder to shut down the server next week to see what breaks. we'd prefer to just make it invisible to other services at first, without deleting the VM, in case we need to turn it back on. but the plan is to make sure accessioning continues to work after the was-thumbnail VM is made unreachable for other services.

[mjgiarlo]

@andrewjbtw do you have a sense of what testing needs to take place for this? I wonder if we should do this during the Thursday 9am-12pm window when @tallenaz will be helping us test hydrus/lyberservices entanglement.

[andrewjbtw]

For the workflows, I think we just need to register and/or update a web archive seed object. It might be a good idea to accession a crawl object too since we're looking for hidden connections, but as far as I can tell the crawl objects aren't directly implicated in the thumbnails, just the seeds.

Other than that, I think we just need to check if there are any unexpected alerts or errors from SWAP. The thumbnail server checks SWAP to generate the thumbnails but I don't know if SWAP ever checks for the thumbnail server.

[mjgiarlo]

@andrewjbtw OK, how do you feel about doing this during the Thursday morning window? If you're open to it, I will reach out to Ops. (I ordinarily wouldn't suggest doing two of these tests at the same time, but I think Hydrus and web-archiving are pretty independent of one another...)

[andrewjbtw]

that works for me

[mjgiarlo]

Cc: @tallenaz, if you're open to it, here's another decommission test we can run on Thursday morning.

[tallenaz]

OK. To clarify, what's the second test? Is it "disassociate was-thumbnail from IP" the checklist above. If so, does "shut down was-thumbnail" count as an interpretation? Looks like there's an explicit "disassociate IP" function in Azure but no straightforward analogue in VMWare.

[mjgiarlo]

@tallenaz I think it'd be fine to shut down the VM, yes, but let me tag prior folks to see if they have any objections: @jcoyne @jmartin-sul @andrewjbtw

[andrewjbtw]

@tallenaz I think shutting it down should be fine.

[mjgiarlo]

@jcoyne @jmartin-sul

I worked with @tallenaz and @andrewjbtw to decommission was-thumbnail-stage and -prod, and we can confirm that all the tests identified above worked just fine. No weird new behavior, and no unexpected exceptions in honeybadger. We will leave the thumbnail service boxes off but leave them around for three weeks, and then re-assess. That way if they are needed in a pinch, we can turn them back on.

@andrewjbtw will proceed with cleaning up stacks soon.

@tallenaz will check back on this in October and if none of us have lingering concerns, he will decommission the VMs for good. At that point, this issue can be closed.

(This is no longer being actively tracked by the @sul-dlss/infrastructure-team FR.)

AFAICT, the remaining was-thumbnail-service tendrils are in documentation and config:

• https://github.com/sul-dlss/shared_configs/tree/was-thumbnail-service-prod
• https://github.com/sul-dlss/shared_configs/tree/was-thumbnail-service-qa
• https://github.com/sul-dlss/shared_configs/tree/was-thumbnail-service-stage
• https://github.com/sul-dlss/was-thumbnail-service (deprecate/archive github repo?)
• https://github.com/sul-dlss/web-archiving/wiki/Glossary

[jmartin-sul]

AFAICT, the remaining was-thumbnail-service tendrils are in documentation and config:
...
* https://github.com/sul-dlss/was-thumbnail-service (deprecate/archive github repo?)

yeah, i'd be in favor of deprecating and archiving once we're sure the VM can be retired