Advice for caching

[Fristi]

Hi there!

Currently building esp32 firmware locally, would like to move this to CI and also to other machines of developers easily by using dagger. This a tool which is based on docker layers and thus can mount cache directories from previous builds.

The build tool will no run a certain step if the inputs didn't change.. e.g. withDirectory("/tmp/build", src) where src is a directory from the host system which includes moving parts like outputs of local builds and such will invalidate the cache each time. If you would provide a constrained filter which makes the build deterministic/hermetic it will cache it and use the layer

Currently I've the following setup

    return dag
      .container({ platform: arch as any })
      .from("rust:1.88-bookworm@sha256:af306cfa71d987911a781c37b59d7d67d934f49684058f96cf72079c3626bfe0")
      .withExec([
        "sh", "-c",
        "apt-get update && apt-get install -y gcc build-essential curl pkg-config"
      ])
      .withMountedCache("/root", dag.cacheVolume("edge-peripheral-root"))
      .withMountedCache("/usr/local/cargo/registry", dag.cacheVolume("edge-peripheral-cargo-registry"))
      .withMountedCache("/usr/local/cargo/git", dag.cacheVolume("edge-peripheral-cargo-git"))
      .withMountedCache("/usr/local/rustup/toolchains/esp", dag.cacheVolume("edge-peripheral-rustup-toolchain-esp"))
      .withMountedCache("/workspace/edge-periphera/target", dag.cacheVolume("edge-peripheral-target-folder"))
      .withExec(["sh", "-c", "cargo install espup --locked --version 0.15.1"])
      .withExec(["sh", "-c", "espup install -t esp32"])

Things which are not cached

• cargo install espup --locked --version 0.15.1 is not backed and being run every time .. it downloads crates and compiles them
• espup install -t esp32 is also being run every time .. it mentions [warn]: Failed to detect version of Xtensa Rust, reinstalling it .. I'm mounting the cache directory of cargo and also the toolchain directory .. but no luck?

[SergioGasquez]

Hi! Sadly, I dont have any experience with Dagger and I cant really help you, feel free to ask in our matrix channel .

An idea to solve the cargo install is to replace it with the binary, maybe try downloading the latest bin for gh release. As for the espup install command, I have no idea tbh, for some reason its not detecting the previous installation.