st2packs-image - docker multi stage build issue - builder stage being missed on MacOS

[PhillypHenning]

Summary

I've run into an issue in which the Docker multi stage build seems to be exhibiting unexpected / undesirable behavior which is causing issues for st2packs usage.

Related Issue from buildkit moby/buildkit#816

In short;
The build instructions for the builder stage are being ignored when using the suggested docker build --build-arg PACKS="<pack names>" -t ${DOCKER_REGISTRY}/st2packs:<version_number> st2packs-image command.

Issue visualized

command
docker build --no-cache --build-arg PACKS="aws" -t awildphil/st2packs:1 st2packs-image --progress plain

build logs

#5 [stage-1 1/1] FROM docker.io/stackstorm/st2packs:runtime@sha256:33f4e77bdd2e942750d8631668ccc9973f12719dac96dc4081e99d30934da69c
#5 sha256:b7b5e5061b2c1f6102a7ceb5a5b42cd04647ab3b4140999a86bf506c600932c2
#5 CACHED

#6 [builder 1/2] FROM docker.io/stackstorm/st2packs:builder
#6 sha256:44de4ae3067832ab1ea2ff2f56e09f71e5584def8b5dc65a88d2a3575cb9ffec
#6 resolve docker.io/stackstorm/st2packs:builder
#6 resolve docker.io/stackstorm/st2packs:builder 0.1s done
#6 CACHED

#7 [stage-1 2/1] COPY --from=builder /opt/stackstorm/packs /opt/stackstorm/packs
#7 sha256:970c29dc5d40e0837a8c2af438f84d250263de882d5c801f4ccfb4022553efd7
#7 DONE 0.1s

#8 [stage-1 3/1] COPY --from=builder /opt/stackstorm/virtualenvs /opt/stackstorm/virtualenvs
#8 sha256:401b19f099d43eee608d293d3d89e8819cb5a91936ef477aad41895932db733c
#8 DONE 0.0s

#9 exporting to image
#9 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#9 exporting layers 0.0s done
#9 writing image sha256:9aaa0ce7359a099526b6af1f8de0cc5420caf3de986c1b1c516a3d23d0555845 done
#9 naming to docker.io/awildphil/st2packs:1 done
#9 DONE 0.1s

container logs

[Bitovi] [st2packs-dockerfiles] $ docker run --rm -it awildphil/st2packs:1 sh
/ # ls /opt/stackstorm/
packs        virtualenvs
/ # ls /opt/stackstorm/packs/
chatops  core     default  linux    packs

As you can see;

1. The st2-pack-install command wasn't invoked by the build
2. The aws pack is not where we expect it.

Fix

By adding the following COPY instructions to the bottom of the Dockerfile we get the expected behavior.

st2pack-image/Dockerfile

... previously committed
FROM stackstorm/st2packs:runtime

COPY --from=builder /opt/stackstorm/packs /opt/stackstorm/packs 
COPY --from=builder /opt/stackstorm/virtualenvs /opt/stackstorm/virtualenvs

command
export DOCKER_BUILDKIT=1; docker build --no-cache --build-arg PACKS="aws" -t awildphil/st2packs:1 st2packs-image --progress plain

build logs

#8 [builder 3/2] RUN /opt/stackstorm/st2/bin/st2-pack-install aws
#8 sha256:5963ceb010683f1dc760d801db7dc8a7ce541630a9a38b543c18bd602cad41be
#8 0.860 2022-11-30 19:06:22,708 INFO [-] Installing pack "aws"
#8 2.964 2022-11-30 19:06:24,812 INFO [-] Successfully installed pack "aws"
#8 2.964 2022-11-30 19:06:24,812 INFO [-] Setting up virtualenv for pack "aws"
#8 2.964 2022-11-30 19:06:24,813 INFO [-] Virtualenv path "/opt/stackstorm/virtualenvs/aws" doesn't exist
#8 12.22 2022-11-30 19:06:34,066 INFO [-] Successfully set up virtualenv for pack "aws"
#8 DONE 12.9s

#9 [stage-1 2/3] COPY --from=builder /opt/stackstorm/packs /opt/stackstorm/packs
#9 sha256:71bfca09a805a8b2ba11a26da5448bc170bf01f9f16bdd446b9d9e73a7aa926a
#9 DONE 0.6s

#10 [stage-1 3/3] COPY --from=builder /opt/stackstorm/virtualenvs /opt/stackstorm/virtualenvs
#10 sha256:8fcf9cbc9a88f30980d86e1268518e913d1cbef4ef37b5eede94e0c70034d9de
#10 DONE 0.7s

#11 [stage-1 4/3] COPY --from=builder /opt/stackstorm/packs /opt/stackstorm/packs
#11 sha256:abbb9ea4e1266b0ba7d1802a4fb9a2c24e865a010a49012291621fa29d37bda2
#11 DONE 0.7s

#12 [stage-1 5/3] COPY --from=builder /opt/stackstorm/virtualenvs /opt/stackstorm/virtualenvs
#12 sha256:94915dd090c49db3f5d15b226f8921d7f2d7fcdbe0cba7f4523980c5ee44ddb0
#12 DONE 1.0s

container logs

[Bitovi] [st2packs-dockerfiles] $ docker run --rm -it awildphil/st2packs:1 sh
/ # ls /opt/stackstorm/packs/
aws      chatops  core     default  linux    packs

Solution 2

command
export DOCKER_BUILDKIT=0; docker build --no-cache --build-arg PACKS="aws" -t awildphil/st2packs:1 st2packs-image --progress plain

Details

Docker Version: Docker version 20.10.7, build f0df350
OS Version: {
Big Sur - v11.6.4
Processor: 2.6 GHz 6-Core Intel Core i7
Memory: 32 GB 2667 MHz DDR4
}

[arm4b]

On Linux, the build instructions work as is.
So yeah, from the issue you mentioned, setting the DOCKER_BUILDKIT=0 fixes it on MacOS.
I'd prefer an ENV as a fix as it'll be less magic.

[alec-afs]

+1 I was having the same problem on macOS with Apple silicon. I added:

COPY --from=builder /opt/stackstorm/packs /opt/stackstorm/packs 
COPY --from=builder /opt/stackstorm/virtualenvs /opt/stackstorm/virtualenvs

to the end of my Dockerfile and this fixed my issue.