Builder instance removal fails on Elastic CI Stack for AWS

[trvrnrth]

When running on Elastic CI Stack builder instances cannot be removed.

The relevant hook actions that cause the issue are as follows:

• Agent environment hook creates isolated docker config
• Plugin pre-command hook creates/uses builder
• Agent pre-exit hook removes isolated docker config orphaning builder containers
• Plugin pre-exit hook cannot remove builder as config no longer exists

Perhaps the builder cleanup could be moved to a plugin post-command hook?

[toote]

@trvrnrth that is indeed an interesting scenario! From what I understand the only reason to use builders would be to isolate a shared docker instance... which is exactly what the elastic stack agent hooks are doing so the usage of builders in that scenario wouldn't be necessary. Or is there any reason to use builders in these situations?

I was thinking about making the change you suggested as well... but then I realized that it is important that it is part of the pre-exit hook to ensure that it is run even when the step is cancelled. There is no way to ensure it will do so otherwise, probably leaving behind those configurations and builders.

[trvrnrth]

@toote My current requirement is to be able to make use of a builder running the docker-container build driver in order to export build cache.

[trvrnrth]

I guess an alternative arrangement would be to have the agent pre-exit hook clean up all builders added to the temporary config before cleaning up those files. Perhaps the plugin could then skip the remove if BUILDKITE_DOCKER_CONFIG_TEMP_DIRECTORY was set and equal to DOCKER_CONFIG or something like that in the knowledge the removal will already have been handled?

As a workaround for the time being I've put repo pre and post command hooks in place to handle builder creation and removal. It's not perfect, but good enough for now as our agents cycle out often enough that dangling containers after cancellation won't be a huge issue. I did also consider a shared builder created at instance boot but in order to use it I'd have to fight against the config isolation instead.

[tigris]

Jumping in here to add

the only reason to use builders would be to isolate a shared docker instance

As @trvrnrth pointed out, if you want to cache multi-stage builds on AWS ECR, you need to create a docker builder and use the docker-container driver. There is no other way.

I tried my best to document my findings here https://gist.github.com/tigris/e601611ac6b7415a49c8e0425cdb22cc

We haven't yet hit this particular issue, but we don't have many builds using customer docker drivers. But we do have a lot of multi-stage builds and have been wondering why cache hits are so low. Now we know, we'll likely hit this issue very soon 😬.