Current layout can cause cvmfs issues with environments.

[marcmengel]

Initial report from @rlcee:

=-=-=-=-=
A few weeks ago, I installed a spack package in mu2e cvmfs and I hit the cvmfs "too many files" error. So in our dirtab:
/cvmfs/mu2e.opensciencegrid.org/.cvmfsdirtab
I added this line:
/spackages//spack/opt/spack///
which solved that problem, but this produces a new catalog file which is inside the individual spack packages:
/cvmfs/mu2e.opensciencegrid.org/spackages/241207/spack/opt/spack/linux-almalinux9-x86_64_v2/gcc-11.5.0/ifdhc-2.8.0-poo3j5ttc5psrctxttxfwikefkolut4u/.cvmfscatalog
I think this is necessary because the parent directory (gcc-11.5.0) contains "all the packages" so it has to be broken up.

Later this led to a new error:
what(): PANIC: /builddir/build/BUILD/cvmfs-2.13.3/cvmfs/sync_item.cc : 294
Error: '/cvmfs/mu2e.opensciencegrid.org/spackages/241207/spack/var/spack/environments/ops-018/.spack-env/._view/uhipw6lewg2rggaejfrjamrnr4xf44st/.cvmfscatalog' is not a regular file.

which I think is spack finding the new file in the package and trying to include it in the spack view. cvmfs rightly complained and I was able to delete the link and proceed. I think this will be OK. However I would rather not be chasing this problem forever. I might be able to block the creation of the link inside spack, which I will look into, but I was wondering if you have any advice from the cvmfs perspective.
Thanks
=-=-=-=-=-=

So the simplest thing, if setting up a new Spack instance for cvmfs, is to add a $SPACK_DIR/etc/spack/config.yaml with:

config:
  install_tree:
    projections:
      all: "{architecture.platform}-{architecture.target}/{name}/{version}-{hash}"

this is different from the default one by a slash between name and version, rather than a dash, which gives another subdirectory layer per package.
But you have to do this before you install anything in the instance, (or else rename all the existing directories and add symlinks with the current names,
which got baked into the binaries when you installed the packages..)
I'm looking at adding this to our Fermilab Spack branch in the default configs so it will get done automatically.

I think we may be able to patch Spack here:

https://github.com/FNALssi/spack/blob/fnal-develop/lib/spack/spack/directory_layout.py#L122

to ignore any ".cvmfs*" files in view merges, etc.