Files missing on rehydration

[LR-JustinPhillips]

Describe the bug

When rehydrating a dependency, some files are not restored. Specifically, if there are multiple files with the same contents, even if they have different paths, only one of them will be hydrated and the rest will be ignored.

Steps to reproduce

Listed in this repo, along with an example.

Expected behavior

All files in the output folder after a normal build should also be present after rehydrating the cached output.

Screenshots

The dist folder of the dependency after a fresh build.

The dist folder of the dependency when rehydrated from the cache.

The console error after the second build.

Environment

System:
OS: Linux 5.15 Ubuntu 22.04.3 LTS 22.04.3 LTS (Jammy Jellyfish)
CPU: (12) x64 AMD Ryzen 5 5600X 6-Core Processor
Memory: 13.05 GB / 15.57 GB
Container: Yes
Shell: 5.1.16 - /bin/bash
Binaries:
Node: 18.19.1 - ~/.nvm/versions/node/v18.19.1/bin/node
npm: 10.2.4 - ~/.nvm/versions/node/v18.19.1/bin/npm
pnpm: 9.15.4 - ~/.nvm/versions/node/v18.19.1/bin/pnpm
Managers:
Apt: 2.4.11 - /usr/bin/apt
Utilities:
Make: 4.3 - /usr/bin/make
GCC: 11.4.0 - /usr/bin/gcc
Git: 2.34.1 - /usr/bin/git
Curl: 7.81.0 - /usr/bin/curl
OpenSSL: 3.0.2 - /usr/bin/openssl
Virtualization:
Docker: 27.5.0 - /usr/local/bin/docker
IDEs:
Nano: 6.2 - /usr/bin/nano
VSCode: 1.96.4 - /home/jphillips/.vscode-server/bin/cd4ee3b1c348a13bafd8f9ad8060705f6d4b9cba/bin/remote-cli/code
Vim: 8.2 - /usr/bin/vim
Languages:
Bash: 5.1.16 - /usr/bin/bash
Perl: 5.34.0 - /usr/bin/perl
Python3: 3.10.12 - /usr/bin/python3

Additional context

This was noticed because we had a project with a file larger than the max blob size that would not get cached; then, when inevitably rebuilding it a second time, the hydrated dependency would miss files that a freshly-built dependency would have.

[LR-JustinPhillips]

Additionally, what happens when you comment in the index files to differentiate them (the index.js files appear):

The build is still broken however, because the index.d.ts files also remove each other. Since those are automatically generated, I can't really comment in them to add differences.

[LR-JustinPhillips]

More evidence: this workaround renames all the index files to match their directory names instead. The build process runs smoothly both times.

[LR-JustinPhillips]

More evidence: this workaround renames all the index files to match their directory names instead. The build process runs smoothly both times.

This strategy actually didn't work on our real monorepo; we had to rename the files the "index" files were referencing so that they would have different contents. I'm not sure how this one ended up working...

[milesj]

@LR-JustinPhillips What does your task config look like? Is this using remote cache?

[LR-JustinPhillips]

The configuration stuff is in the repo, but I'll copy it over for you:

.moon/workspace.yml

projects:
  - "projects/**/moon.yml"

vcs:
  defaultBranch: "main"
unstable_remote:
  cache:
    compression: "zstd"
  host: "grpc://some.domain.name:9099" # remember to change this

.moon/tasks/node.yml

fileGroups:
  configs:
    - "*.json"
    - "*.config.ts"
    - "/pnpm-lock.yaml"
  sources:
    - "src/**/*"
    - "index.html"
  assets:
    - "public/**/*"
  tests:
    - "tests/**/*"
    - "**/__tests__/**/*"

tasks:
  build:
    script: "pnpm run build"
    deps:
      - "^:build"
    inputs:
      - "@files(configs)"
      - "@files(assets)"
      - "@files(sources)"
    outputs:
      - "dist"

To answer your question, yes, this is using remote caching; we are using the zstd option, if that makes any difference.

[milesj]

There's some bugs in remote caching right now, so I'm leaning towards the uploads failed at some point, so they will never rehydrate because they don't exist.

v1.32 fixes a ton of remote caching bugs, so I'd suggest turning it off for now.

[milesj]

1.32 out, give that a try. Has lots of fixes.