Repo hooks are executed outside k8s job

[keith]

When using buildkite's repo hooks as described here https://buildkite.com/docs/agent/v3/hooks#available-hooks if you have a post-checkout or pre-command hook, they are run outside of the k8s job that actually ends up running your code. This fails in a few ways for us since we use those to do some setup that depends on tools that are in our custom docker image, and to export environment variables we expect to exist later.

I believe this is because the core command is not run by buildkite-agent but instead is run directly

k8s-buildkite-plugin/lib/job.jsonnet

Lines 285 to 286 in 5826cbd

	command: ['/bin/sh', '-c'],
	args: [env.BUILDKITE_COMMAND],

I'm sure there was a good reason but I wonder if we could somehow run this with the buildkite-agent bootstrap command, which I assume would fix the issue?

k8s-buildkite-plugin/lib/job.jsonnet

Line 321 in 5826cbd

args: ['bootstrap', '--experiment=git-mirrors', '--git-mirrors-path=/git-mirrors', '--ssh-keyscan', '--command', 'true'],

[keith]

Note that buildkite-agent bootstrap supports --phases to do potentially less work. This could mean, depending on the background of that container being separate, we could change that command to run less and also run the second command through buildkite-agent

[iffyio]

I'm wondering if this is already possible, by default the buildkite agent binary is available to the job, thinking it should be possible to run bootstrap as the job's command instead - would those work in this case?

[keith]

Interesting! So I guess we could override that nice, I'm not sure if that's still how it should work though?

[iffyio]

Yeah it might not be, I'd say we give it a try first and if it'll be significantly simpler to have the plugin run via buidlkite-agent we can look into that - I can't really think of any disadvantages atm with running via the agent beyond the command's logs including buildkite output which should be fine (especially if we make it an opt-in feature)