MountTmpfsAtTemp: Option to mount tmp on ephemeral mount

[pquerna]

Is your feature request related to a problem? Please describe.
We have some jobs which use a lot of tmp -- too much for tmpfs in memory. So, we have set MountTmpfsAtTemp=false. This uses keeps /tmp on the root device (EBS).

But, we are also using instance types with ephemeral NVMe disks (EnableInstanceStorage=true).

Describe the solution you'd like
If EnableInstanceStorage=true and MountTmpfsAtTemp=false, it would be great to mount tmp on the /mnt/ephemeral mount. Or another configuration option to opt specifically into this behavior.

Additional context

It looks like most of this functionality is in https://github.com/buildkite/elastic-ci-stack-for-aws/blob/main/packer/linux/conf/bin/bk-mount-instance-storage.sh#L27-L42 -- might need somre re-ordering with the script, but should be pretty feasible?

[dabarrell]

Hey @pquerna! Thanks for raising this – we can see the value in what you're suggesting. Would you be up for submitting a PR to make this change?

One option that could be explored is symlinking /tmp to /mnt/ephemeral/tmp or similar. It'll need to be behind a parameter as well – no good names come to mind, so feel free to suggest one!