Handle empty directories

[kshitijl]

Hi! I'd like to use filepack as part of my backup process. I'm glad I found it because it's exactly the thing I wanted to write.

Some of the directories I'm backing up have empty subdirectories from various builds or downloads that have happened there. I'd still like to use filepack for these. I'd be happy to write the code for it and submit a pull request, but I wanted to open this issue to see if that's something you'd be interested in.

Options:

1. Just ignore empty directories and don't put them in the manifest. This has the downside of being surprising: two directories can have different contents (in terms of empty directories) but the same manifest.
2. Put empty directories in the manifest with size 0 and whatever blake3 produces for hashing null. This has the downside of putting a directory in the manifest when everything else is files.
3. A separate empty_directories section which lists them. This is clean and explicit, but makes the manifest schema a little more complicated. This field should be omitted when there are no empty directories.

Let me know if any of these or some other option sounds like something you'd support. Again, I'd be happy to implement it and send a PR.

Why not just delete empty subdirectories?

Most importantly, some of them are inside the .git subdirectory, like refs/heads and refs/tags, and I'm not sure what the semantics are for just deleting them. I can imagine other tools that use the existence or nonexistence of empty subdirectories, or that create a subdirectory at some initialization point and then expect it to be there later.

[repi]

Ran into this problem also when using filepack as part of backup integrity checking and Cryptomator can leave empty folders (related old discussion: cryptomator/cryptomator#1749) which filepack fails then to create the manifest for the entire folder, which does feel a bit harsh.

$ filepack create test-backup
error: empty directories `d\BD`, `d\EJ`, `d\FI`, `d\IR`, and `d\SL`

I would suggest the empty directories are included in the manifest to be the most inclusive and explicit by default, so something like option 3 that @kshitijl suggested. And possibly having some argument for ignoring empty directories like --ignore-empty-dirs for users that do not want them to be part of the manifest or the verification of the manifest.

Probably if one has generated with --ignore-empty-dirs that option should be included in the manifest so the verification of the manifest also ignores empty directories when comparing.

Although I appreciate that would add extra complexity. My specific use case I may be able to solve with just nuking empty directories, though there could be other use cases where they are important (as mentioned on above poster)

[casey]

Thanks for opening this issue! I agree that it's a problem, and that empty directories should be handled better. Ideally, I think empty directories should somehow be added to the manifest, and verified when verifying.

The way that I'd ultimately like to handle this is to make the manifest contain a recursive dictionary of directories and files, with each entry key being a filename. This way, empty directories can be represented simply as empty dictionaries.

I haven't actually thought about how this would be represented in the manifest, but something like:

{
  "entries": {
    "a": { "entries": {} },
    "b": {
      "size": 124,
      "hash": "787038a2c28ef851bbc8280478304cf1b03b56802cf6a580f095126eb289ebd1",
    }
  }
}

In this case, the top-level directory contains an empty directory named a, and file named b with size 124 and hash "787038a2c28ef851bbc8280478304cf1b03b56802cf6a580f095126eb289ebd1".

If a dictionary contains an entries member, it is a directory, and if if contains as size and hash member, it is a file.

This would also entail changing how the root fingerprint is calculated.

However, all the above being said, I'm not actively working on filepack at the moment, so I don't know when I'll get to any of this.

[kshitijl]

Your proposal sounds good to me. Would you entertain a PR? I'm happy to implement the thing you said.