evaluate materials backup strategy

[koddo]

Any particular reason not to use tar+gpg for backups? LUKS seems error-prone compared to a single command of encrypting a folder.

Thank you for the guide, by the way.

[drduh]

Indeed, one could symmetrically encrypt the private key and backup contents with GPG (though I would still recommend using a different passphrase from identity). The resulting archive would be more portable and convenient to use.

However, portability and convenience of accessing backups is not a goal of this guide and, in a way, LUKS limits portability to encourage use of a secure-ish operating environment. I also suppose the argument could be made that using different encryption programs to secure key material is "better", but in practice this is likely not relevant. In other words, in terms of at-rest encryption protection, LUKS and GPG are likely equivalent.

I'm curious what you mean by LUKS being error-prone. If it truly is a hassle for readers, we could simplify the instructions and move LUKS to Optional Hardening. What do you think? Is there anything else to consider?

[koddo]

Maybe it's just me, but I never interact with LUKS directly, which means I don't know what all those commands mean, and there's a lot to learn already about gpg and yubikey-manager. Also, by error-prone I mean manipulation of partitions in /dev manually: commands like sudo dd if=/dev/zero of=/dev/sdc bs=4M count=1 seem dangerous for sleep-deprived users, I didn't mess up purely by luck. Everything else is recoverable.

[forbytten]

Another option may be to use LUKS with a loop device, the latter of which is a file used as a block device, not to be confused with loop-AES, the latter of which seems to be an alternative to LUKs. Using a loop device has similar advantages to the current LUKs steps but has two, IMHO, significant advantages:

1. Can be safely automated, addressing koddo's concerns. All destructive operations are restricted to a file, not an entire partition or disk. This is the approach I plan to try with the automation scripts I'm currently implementing for my use case.
2. It's portable. Being able to back up data remotely may be recommended, depending on a user's specific security threat model, as well as their physical environment, such as living in a natural disaster prone region.

Disadvantages:

1. If a user chooses not to use an ephemeral environment and they forget to delete the encrypted file after copying it to portable devices and/or remote storage, the file may be more vulnerable to exfiltration than the current physical disk based approach but this threat seems minor compared to the advantages and the file is encrypted.
2. Nothing else obvious or significant that I can think of. However, I have minimal experience with LUKS and loop devices so it would be great if someone more experienced could comment on this approach.

I think this approach can also be used with the OpenBSD equivalent, virtual node devices, but I have not tried it.

Linux Steps

The steps have been tested on Debian Live 12.8.0:

009f482430b8505bba8099aeac3f59eab92ddfa30ae8910d6293f7e198194bad24fa625fcfa38813e6e25b2b7502a542fa04af4e4c2a52376c36162fea8debf7  debian-live-12.8.0-amd64-gnome.iso

user@debian:~$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
IMAGE_ID=live
BUILD_ID=20241109T101058Z

The loop device specific steps are based on the cryptsetup FAQ:

zcat /usr/share/doc/cryptsetup/FAQ.md.gz|grep -A 10 '2.6 How do I use LUKS with a loop-device?'
  * **2.6 How do I use LUKS with a loop-device?**

  This can be very handy for experiments.  Setup is just the same as with
  any block device.  If you want, for example, to use a 100MiB file as
  LUKS container, do something like this:
```
    head -c 100M /dev/zero > luksfile               # create empty file
    losetup /dev/loop0 luksfile                     # map file to /dev/loop0
    cryptsetup luksFormat --type luks2 /dev/loop0   # create LUKS2 container
```
  Afterwards just use /dev/loop0 as a you would use a LUKS partition.

I have marked the new/modified steps with (changed). All the other steps are existing from the guide.

(changed) Choose a name for the encrypted loop device file:

export LUKS_GPG_BACKUP_FILE="gpg_backup_luks"

(changed) Create a small (at least 20 Mb is recommended to account for the LUKS header size) loop device file for storing secret materials:

head -c 20M /dev/zero > "$LUKS_GPG_BACKUP_FILE"

(changed) Associate $LUKS_GPG_BACKUP_FILE with the next available loop device, automatically ensuring the file isn't mapped to multiple loop devices:

sudo losetup -f -L "$LUKS_GPG_BACKUP_FILE"

(changed) Ensure losetup is on non-sudo user PATH, which is needed on some OS's such as Debian Live 12.8.0:

export PATH="/usr/sbin:$PATH"

(changed) Determine what loop device was used:

export LUKS_GPG_BACKUP_LOOP_DEVICE=$(losetup -j "$LUKS_GPG_BACKUP_FILE" |cut -d: -f1)

Generate another unique Passphrase (ideally different from the one used for the Certify key) to protect the encrypted volume:

export LUKS_PASS=$(LC_ALL=C tr -dc 'A-Z1-9' < /dev/urandom | \
  tr -d "1IOS5U" | fold -w 30 | sed "-es/./ /"{1..26..5} | \
  cut -c2- | tr " " "-" | head -1) ; printf "\n$LUKS_PASS\n\n"

This passphrase will also be used infrequently to access the Certify key and should be very strong.

Write the passphrase down or memorize it.

(changed) Format the loop device:

echo $LUKS_PASS | sudo cryptsetup -q luksFormat "$LUKS_GPG_BACKUP_LOOP_DEVICE"

(changed) Mount the loop device

echo $LUKS_PASS | sudo cryptsetup -q luksOpen "$LUKS_GPG_BACKUP_LOOP_DEVICE" gnupg-secrets

Create an ext2 filesystem:

sudo mkfs.ext2 /dev/mapper/gnupg-secrets -L gnupg-$(date +%F)

Mount the filesystem and copy the temporary GnuPG working directory with key materials:

sudo mkdir /mnt/encrypted-storage

sudo mount /dev/mapper/gnupg-secrets /mnt/encrypted-storage

sudo cp -av $GNUPGHOME /mnt/encrypted-storage/

(changed) An example to better visualize the relationship between the underlying file gpg_backup_luks, the loop device and the LUKS container:

user@debian:~$ losetup -l -j gpg_backup_luks
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE                  DIO LOG-SEC
/dev/loop1         0      0         0  0 /home/user/gpg_backup_luks   0     512

user@debian:~$ lsblk -p /dev/loop1
NAME                        MAJ:MIN RM SIZE RO TYPE  MOUNTPOINTS
/dev/loop1                    7:1    0  20M  0 loop
└─/dev/mapper/gnupg-secrets 253:0    0   4M  0 crypt /mnt/encrypted-storage

(changed) Unmount and close the encrypted volume and detach the loop device:

sudo umount /mnt/encrypted-storage

sudo cryptsetup luksClose luksfile

sudo losetup -d "$LUKS_GPG_BACKUP_LOOP_DEVICE"

(changed) Copy $LUKS_GPG_BACKUP_FILE to multiple portable storage devices and potentially remote storage.
(changed) If you are not using an ephemeral OS, delete the local copy.

Signing

I'd also suggest with this approach to also sign the backing file. This would be especially useful if backing up the file to remote storage.

[drduh]

@forbytten your approach is novel and perhaps technically superior, but I understand the original request as wanting to reduce overall complexity during setup.

can we do this without also materially impacting the security and durability of materials?

[forbytten]

You're right that I focused only on dd being dangerous, due to confirmation bias, as dd was my main concern. I'll leave it up to @koddo to comment on whether they view complexity as having the same "badness" weight as dd being dangerous and error prone but in general, I would say that simplicity is best.

The available backup options appear to fall within the falling categories:

1. tar + encrypt, such as tar + GPG. Sub-categories:

   • public key encryption
   • symmetric encryption

2. filesystem level encryption. Sub-categories:

   • native, supported by the filesystem itself, such as ext4 + fscrypt
   • stacked, layered on top of the real filesystem. Probably little reason to consider these if a native solution exists.

3. block device level encryption, such as LUKS2. Sub-categories:

   • physical block device (existing YubiKey Guide approach)
   • virtual block device (the loop device approach I suggested)

To elaborate on the ext4 + fscrypt option, I think it is simple to use and I believe provides similar security benefits as the existing approach but does not require the user to create a new partition. However, it does assume they already have an ext4 filesystem to create the encrypted directory on. Some links to the basic usage:

1. Ensure encryption is enabled on the ext4 filesystem

2. Set up fscrypt on empty directory

3. Lock and unlock the directory when you want to close/open it

As with many things in security, I don't think there is a single "best" option so I've had a go at a pro/con matrix below, ignoring stacked filesystems. I suggest the "right" choice depends on which criteria you view as taking priority for the guide. However, whatever choice is made - be it the current approach or something new - perhaps something like this matrix could be included in the guide to allow users to explore other options if they aren't comfortable with the recommendation, similar to how Prepare environment covers various options?

Criteria	tar + GPG public key encryption	tar + GPG symmetric encryption	ext4 + fscrypt	LUKS2 + physical device	LUKS2 + loop device
Usage complexity (Low/Medium/High)	Low	Low	Medium	Medium	High1
Resistance to offline attacks (Low/Medium/High)	Medium2	High3	High4	High5	High6
Resistance to image exfiltration (Low/Medium/High)	Medium7	Medium7	High8	High	Medium7
Resistance to plaintext persistence (Low/Medium/High)	Low9	Low9	High10	High10	High10
Resistance to file corruption11	Dependent on hosting filesystem	Dependent on hosting filesystem	Ext4 journalling	Dependent on encrypted filesystem	Dependent on both hosting filesystem and encrypted filesystem
Multiple backups	Copy encrypted file locally or remotely12	Copy encrypted file locally or remotely12	Mount, copy files to independently encrypted target13	Mount, copy files to independently encrypted target	Copy encrypted file locally or remotely12
Authenticated encryption	Via GPG -sign option	Via GPG -sign option	No14	No15	Not directly but you can sign the loop device file using GPG. However, having a separate step is prone to inconsistency if you forget to re-sign after changing the data.
YubiKey integration	Encrypt, decrypt, sign	Sign	No	Encrypt/decrypt master key via systemd-cryptenroll16	Encrypt/decrypt master key decrypt via systemd-cryptenroll16
Cross platform access	tar + gpg available on most platforms	tar + gpg available on most platforms	TBD17	TBD17	TBD17
Safe for automation	Yes	Yes	Mostly - some filesystem wide changes are required, such as enabling encryption using tune2fs	No - danger of destroying drive partitions	Yes

Footnotes

1. Requires additional knowledge of loop device management ↩

2. GPG public key encryption uses a hybrid cipher (aka. envelope encryption). The data is encrypted using symmetric encryption, then the symmetric key is encrypted using public key encryption. Of the two encryption types, public key encryption is probably the weaker of the pair. ↩

3. Assuming GPG has been hardened as per the YubiKey Guide. ↩

4. fscrypt does not encrypt "non-filename metadata, such as timestamps, the sizes and number of files, and extended attributes" but for this use case, I'd consider the metadata leakage to be insignificant, given the limited type of data being encrypted. fscrypt defaults to (AES-256-XTS, AES-256-CBC-CTS) for (contents, filename) encryption, the former of which matches LUKS2. AES-256-XTS uses a 512 bit long master key, with only half of the key being used for the underlying AES-256 cipher, the same as the defaults used by cryptsetup. ↩

5. On Debian Live 12.10.0, cryptsetup 2.6.1 defaults to aes-xts-plain64 with a 512 bit long master key, with only half of the key being used for the underlying AES-256 cipher, the same default used by fscrypt:

   user@debian:~$ sudo cryptsetup --version
   cryptsetup 2.6.1 flags: UDEV BLKID KEYRING KERNEL_CAPI
   user@debian:~$ sudo cryptsetup --help |tail -5
   Default compiled-in device cipher parameters:
           loop-AES: aes, Key 256 bits
           plain: aes-cbc-essiv:sha256, Key: 256 bits, Password hashing: ripemd160
           LUKS: aes-xts-plain64, Key: 256 bits, LUKS header hashing: sha256, RNG: /dev/urandom
           LUKS: Default keysize with XTS mode (two internal keys) will be doubled.
   

   ↩

6. I asked on the crypsetup mailing list about the security of LUKS2 on a loop device and Milan Broz, co-contributor of the cryptsetup FAQ (https://gitlab.com/cryptsetup/cryptsetup/-/wikis/FrequentlyAskedQuestions#a-contributors) confirmed it's equivalent to a physical device in terms of the encryption robustness: https://lore.kernel.org/cryptsetup/2ae83c1d-4d1a-4fb6-a46c-0aaa1e024269@gmail.com/ ↩

7. Tar files and a loop device file are only considered easier to exfiltrate in the sense they more easily afford the end user accidentally leaving the files lying around somewhere they shouldn't be and facilitating remote access, as opposed to a physical USB drive. However, human error could also result in the USB drive being mounted long term so it's maybe only a minor distinction. ↩ ↩² ↩³

8. ext4 + fscrypt directories cannot be copied in their encrypted form, only when mounted, and are thus as resistant to image exfiltration as LUKS2 + physical device. If you attempt to copy the locked dir or anything in it, an error will occur:

   ```
   cp: cannot open 'encrypted-dir/mKsr5Y8OVRkIHKTsT3L25O2_K-pf7hntVBr3yUeuwOGyUZGZ5S11ig' for reading: Required key not available
   ```
   

   ↩

9. GPG decryption will persist plaintext to disk, and is thus vulnerable to plaintext persistence if the underlying disk is physical and not encrypted. Furthermore, even if the underlying disk is encrypted, the encryption strength is independent of the GPG encryption and could be weaker. Accidental plaintext persistence in this context is not totally disastrous, though, as the private keys are still encrypted by gpg-agent. However, gpg-agent only uses AES-128, which is expected to be vulnerable to Quantum Computing, if that ever eventuates. Furthermore, the GPG revocation key is plaintext. ↩ ↩²

10. fscrypt and LUKS2 transparently decrypt data as it is read so the plaintext is never persisted to disk. ↩ ↩² ↩³

11. None of the options dictate the hosting or encrypted filesystem, except for the ext4 + fscrypt option which is specifically using ext4's fscrypt support. However, in all cases, a modern journaled filesystem such as ext4 is recommended. ↩

12. Depending on the context, the ability to easily backup the encrypted data to a remote location may be a positive or negative. For example, remote backups may prove useful in locations with questionable physical security or in disaster prone areas, in which case the benefit may outweigh the negative of potential exfiltration. ↩ ↩² ↩³

13. fscrypt - Backup, restore, and recovery ↩

14. fscrypt supported modes:

    Authenticated encryption modes are not currently supported because of the difficulty of dealing with ciphertext expansion

    ↩

15. LUKS2 has experimental support for authenticated encryption, listed in the cryptsetup man page, but EXPERIMENTAL and backup do not seem like a good match:

    Authenticated disk encryption (EXPERIMENTAL)
    Since Linux kernel version 4.12 dm-crypt supports authenticated disk encryption.

    ↩

16. In my testing, systemd-cryptenroll uses PBKDF as the key derivation function when enrolling a FIDO2 device with LUKS2, whereas LUKS2 uses argon2id for passphrases. If the entropy of the FIDO2 generated credential is truly high, this shouldn't matter but I would probably feel safer if argon2id is used as a defence-in-depth measure against FIDO2 implementation weaknesses. If in doubt, I'd recommend choosing a high entropy user chosen passphrase instead and forgo the ease of use of YubiKey integration. ↩ ↩²

17. I don't have convenient access to a Mac to test on and I only have limited experience with BSDs and WSL ↩ ↩² ↩³

[koddo]

@forbytten, this is a great summary, thank you.

A couple of questions:

1. When a user encrypts and decrypts backups while running a linux live image, the risk of plaintext persistence is mitigated, as everything is on in-memory filesystem, right?

2. What is image exfiltration? And why the risks are higher in the case of tar+gpg? I couldn't find anything on this myself.

[forbytten]

1. Yes. There's nothing inherently wrong with the tar + GPG approach, just that it relies on the user following correct procedure rather than the underlying tech protecting the user from themselves.

2. For disks, I just mean the ability of an attacker to dump the encrypted blocks of the disk (the image) and transfer them to an attacker controlled destination (exfiltrate) as a precursor to an offline attack. For tar + GPG, I'm loosely generalizing the term for simplicity, treating the encrypted tar file as analogous to the encrypted disk image, even though they are technically very different.

   The data can be considered as being cold (rarely accessed) and thus its natural, long term state for each option is:

   	tar + GPG public key encryption	tar + GPG symmetric encryption	ext4 + fscrypt	LUKS2 + physical device	LUKS2 + loop device
   cold state	encrypted file	encrypted file	encrypted directory	encrypted partition	encrypted file

   In terms of "Resistance to image exfiltration", I just mean that an encrypted file is more easily copied from source to target, whereas for the other two options, to copy the encrypted state requires:

   1. ext4 + fscrypt: as noted in footnote 8, it can't be done by directory or file copy. So you have to dump the raw bytes of the drive, say using dd.
   2. LUKS2 + physical device: similarly, you have to dump the raw bytes of the drive.

   So say we assume the following insecure configuration due to human error: for each option, the encrypted data is stored on a thumb drive and the user accessed it on their daily system instead of an ephemeral system and the user has forgotten to remove it so it's attached long/longish term (but not decrypted). Further assume that for the encrypted file options, the thumb drive itself is not encrypted and is mounted. For these options, a remote attacker only needs file traversal capabilities in order to obtain the encrypted file, whereas for the other two options, the attacker needs RCE (remote code execution) capability, plus typically root permissions I believe, in order to dump the raw bytes of the drive, which are much more stringent requirements.

   Similar to your first question, as long as your operational security practices are fine, IMHO there's no significant issue with the encrypted file options - I am personally partial to LUKS2 + loop device. It's just a criteria I thought worth mentioning in the table. I should have also mentioned the criteria are not sorted in terms of relative importance, as that's mostly context dependent.

[koddo]

@forbytten, thank you for the explanation and analysis.

[forbytten]

I would suggest another criterion to be KDF (key derivation function) support. The KDF is a function that transforms/derives the real symmetric encryption key from the passphrase. It's utility is two fold in this context:

1. Produce a key that has higher entropy than a source which potentially has lower entropy.
2. Be a password hashing function so that it's expensive to compute, in order to make brute forcing the password computationally infeasible. OWASP has a decent primer on password hashing.

Comparing the backup options:

• GPG public key encryption: KDF isn't applicable, exchanging the whole concept for the security of the public key encryption, as the public key is used to encrypt the symmetric key, the latter of which I believe GPG will securely generate with high entropy.

• GPG symmetric encryption

  • The latest OpenPGP spec supports Argon2id. Proton's blog back in Sep, 2023 also mentions that Argon2id is preferable.
  • However, GPG has not added support for Argon2id based on its NEWS page and therefore only supports the old and broken "Iterated and Salted S2K".

• fscrypt has two components:

  • kernel level. Does not use a KDF, instead delegating the responsibility to the userspace tool.
  • userspace tool. Uses Argon2id, although it's unclear if the default computation time of 1 second can be overridden.

• LUKS2 defaults to Argon2id these days. Computation time can be increased via the cryptsetup luksFormat --iter-time option. For example, if you specify 5000 milliseconds, cryptsetup will choose parameters that result in one iteration taking about 5 seconds to compute on your hardware.

Now, the KDF only adds utility if your passphrase is weak or relatively weak. So if you decide on using GPG symmetric encryption to encrypt your backup, as @koddo seems to be, I would say it's even more important to choose a high entropy passphrase. 256 bits of entropy would be ideal, IMHO, matching the AES-256 key size and completely removing the KDF from the threat model.