Rework ChaCha_RNG

[thillux]

This is currently WIP. Will squash when ready.

When reading the source code of the ChaCha_RNG I saw some room for improvement:

• style: use initialization lists
• seeding: use HMAC(SHA-512) instead of HMAC(SHA-256) for less effects of the injectivity of hash functions
• internal state: add optional chacha key overwrite after each operation in order to provide backtracking resistance
• internal state: also use IV/nonce to widen effective key material

@randombit If interested, can you provide me your go program, which created the KAT for the RNG?

I'll adapted KAT test results via copy paste from CLI output and can also generate them externally if needed.

[thillux]

CC @reneme

[thillux]

@reneme Thanks for your review, I probably addressed everything mentioned. I left your suggestions open for you to compare and close, as I did not apply them 1:1. My code also uses the IV/Nonce of ChaCha20 now to extend the key material. I therefore centrally check the length assertion in the suggested truncate function (now a split into two sub-key chunks). I needed some additional variables typically named key_material, otherwise e.g. m_hmac->final() would be vanished before the spans taken from it are used. Tests did not deliver reproducible known answers with your suggestion applied 1:1.

[thillux]

Just FYI speed comparison on AMD Ryzen 9 5950X with different buffer sizes (motivating default disable of fast key erasure):

4 Byte

ChaCha_RNG generate buffer size 4 bytes: 12.029 MiB/sec 269.56 cycles/byte (6.06 MiB in 503.99 ms)
ChaCha_RNG (with key erasure) generate buffer size 4 bytes: 5.607 MiB/sec 578.24 cycles/byte (2.81 MiB in 501.56 ms)

8 Byte

ChaCha_RNG generate buffer size 8 bytes: 24.119 MiB/sec 134.44 cycles/byte (12.06 MiB in 500.13 ms)
ChaCha_RNG (with key erasure) generate buffer size 8 bytes: 11.219 MiB/sec 289.01 cycles/byte (5.62 MiB in 501.37 ms)

16 Byte

ChaCha_RNG generate buffer size 16 bytes: 47.715 MiB/sec 67.96 cycles/byte (23.88 MiB in 500.37 ms)
ChaCha_RNG (with key erasure) generate buffer size 16 bytes: 22.178 MiB/sec 146.20 cycles/byte (11.12 MiB in 501.62 ms)

32 Byte

ChaCha_RNG generate buffer size 32 bytes: 93.908 MiB/sec 34.53 cycles/byte (47.00 MiB in 500.49 ms)
ChaCha_RNG (with key erasure) generate buffer size 32 bytes: 44.689 MiB/sec 72.56 cycles/byte (22.38 MiB in 500.68 ms)

64 Byte

ChaCha_RNG generate buffer size 64 bytes: 181.763 MiB/sec 17.84 cycles/byte (90.94 MiB in 500.31 ms)
ChaCha_RNG (with key erasure) generate buffer size 64 bytes: 89.538 MiB/sec 36.21 cycles/byte (44.81 MiB in 500.49 ms)

512 Byte

ChaCha_RNG generate buffer size 512 bytes: 1035.250 MiB/sec 3.13 cycles/byte (517.62 MiB in 500.00 ms)
ChaCha_RNG (with key erasure) generate buffer size 512 bytes: 588.339 MiB/sec 5.51 cycles/byte (294.19 MiB in 500.03 ms)

1024 Byte

ChaCha_RNG generate buffer size 1024 bytes: 1590.335 MiB/sec 2.04 cycles/byte (795.19 MiB in 500.01 ms)
ChaCha_RNG (with key erasure) generate buffer size 1024 bytes: 1001.902 MiB/sec 3.24 cycles/byte (501.00 MiB in 500.05 ms)

All taken from ./botan speed RNG --buf-size=XXX

[thillux]

Ready from my side, will squash, when review is done.

[coveralls]

coverage: 90.673% (+0.005%) from 90.668%
when pulling 92ae192 on thillux:mtheil/chacha_rng_fke
into 0487ba5 on randombit:master.

[reneme]

@thillux I didn't realize I am authorized to push into your fork 😨 I meant to push into my own branch, but a sole git push -f was a bit too vigorous. Please make sure I didn't accidentally crush something. Sorry about that.

---

Changes itself look good to me and I think the improvements make sense. Thanks!

To the best of my understanding, the ChaCha RNG isn't standardized in any way, so we should have the liberty to improve the implementation as suggested. However, depending on how pedantically we want to follow semantic versioning, these changes might be seen as a violation. E.g. if someone based their unit tests on a hard-coded seed for this RNG their tests would break (much like Botan's KATs had to be updated).

As far as I can see, "breaking" changes are:

• "HMAC(SHA-256)" replaced by "HMAC(SHA-512)"
• derivation and usage of an IV for the internal ChaCha updates

@randombit What's your take on that? Do we want to guarantee stable DRNG outputs across minor versions?

[thillux]

Your push did not break things for me :). I'll squash together now.

[thillux]

@reneme Squashed everything into two commits (change DRNG + tests).

[reneme]

Changes look good to me, apart from the semver concerns and a typo.

Oh, and please note and update the related documentation in rng.rst

[thillux]

I updated the documentation.

[thillux]

While there, I saw, that ESDM_RNG is not documented there. Can I do it in this PR or open another one?

[reneme]

Can I do it in this PR or open another one?

Please open a separate one.

[randombit]

I am a bit hesitant to change the output bytes here since (in a Hyrum's Law sense) the ChaCha_RNG output has been stable since introduction and hypothetically someone might be depending on this. That said

• The docs never guaranteed that it was stable over time
• Applications relying on this kind of RNG stability are relatively uncommon
• The library doesn't generally do a good job of respecting RNG stability anyway (outside of a few specific cases like RSA and ECDSA where we test that it works)

So probably fine to do.

I would request that the doc update note that a) the output changed in 3.10 b) the output might change again at any future time, and if RNG stability is required to use HMAC_DRBG instead.

[I still do want to review this PR fully before merging]

[thillux]

@randombit I added some more notes on output stability as requested.