Requesting a meeting to discuss plans with Wycheproof

[bleichenbacher-daniel]

I would like to have a meeting that discusses the plans of Wycheproof.

In its current state there is a lot of uncertainty and confusion. For example the fate of the generator code for the test vectors is unknown.
As a result I'm rewriting a lot of code that would otherwise not be necessary. The format for the test vectors is unclear. Proposals I made some time ago have not been discussed so far. And it would be a good time to do so now. Communications is poor, which also leads to replication and slow down.
My work in progress is currently here https://github.com/bleichenbacher-daniel/Rooterberg
This effort involves a lot of duplication. It would be very useful for me to sync this and know how to avoid further duplication.

A meeting would in my opinion be a good way forward to understand what has gone wrong with the project and how to fix this.

[J08nY]

I support this. While I understand the move of this project towards more "data-only" test-vectors, there is a lot to be lost here. While git history keeps the test code, it is much harder to go through it. Now, even the "Hall of Bugs" in the README contains broken links, like:

Bouncy Castle's ECDHC leaks private keys: Test: testModifiedPublic and testWrongOrderEcdhc in EcdhTest.

While the tests may have been outdated and testing only a few libraries they could provide great inspiration for testing other libraries that data-only test vectors do not give.

Similarly for the test generation code. If that is not properly open-sourced it will be a pretty bad loss. For me, the priority of "linting of test-vectors" <<< "generation code is open-sourced". I understand the former is important for consumability of the test-vectors, without the latter you will be reinventing the wheel.

@FiloSottile @cpu

[cpu]

While git history keeps the test code, it is much harder to go through it. Now, even the "Hall of Bugs" in the README contains broken links

I've tried to address both points with #139, thanks for flagging.

Similarly for the test generation code. If that is not properly open-sourced it will be a pretty bad loss. For me, the priority of "linting of test-vectors" <<< "generation code is open-sourced".

I have zero insight into the former or current state of the generation code and don't personally see any way to make progress on that front. My perspective is that it doesn't make sense to continue to do nothing waiting for something outside of my control that may/may not happen. I'd prefer to prioritize pragmatic changes that can be done today, accepting the risk some of the work may become redundant in the future.

[J08nY]

While git history keeps the test code, it is much harder to go through it. Now, even the "Hall of Bugs" in the README contains broken links

I've tried to address both points with #139, thanks for flagging.

Sure thing.

Similarly for the test generation code. If that is not properly open-sourced it will be a pretty bad loss. For me, the priority of "linting of test-vectors" <<< "generation code is open-sourced".

I have zero insight into the former or current state of the generation code and don't personally see any way to make progress on that front. My perspective is that it doesn't make sense to continue to do nothing waiting for something outside of my control that may/may not happen. I'd prefer to prioritize pragmatic changes that can be done today, accepting the risk some of the work may become redundant in the future.

Hmm, I see, I thought there was some communication/continuity during the move from Google to C2SP. Do you have any insight on who to bug to make the code release happen?

[bleichenbacher-daniel]

While git history keeps the test code, it is much harder to go through it. Now, even the "Hall of Bugs" in the README contains broken links

I've tried to address both points with #139, thanks for flagging.

Sure thing.

Similarly for the test generation code. If that is not properly open-sourced it will be a pretty bad loss. For me, the priority of "linting of test-vectors" <<< "generation code is open-sourced".

I have zero insight into the former or current state of the generation code and don't personally see any way to make progress on that front. My perspective is that it doesn't make sense to continue to do nothing waiting for something outside of my control that may/may not happen. I'd prefer to prioritize pragmatic changes that can be done today, accepting the risk some of the work may become redundant in the future.

There are of course issues like #106 that can be discussed without having to wait for any response by Google. The new version of the test vectors currently use the proposed format. It seems suitable to me, but maybe someone has some feedback here.

Hmm, I see, I thought there was some communication/continuity during the move from Google to C2SP. Do you have any insight on who to bug to make the code release happen?

On the contrary there was a distinct lack of communication. For example I wasn't even informed about the move for almost one year. Unfortunately, this situation continues to be the case.
Regarding the generation code: I have no idea why previous attempts to get access to the code failed. I have contacted a few people at Google, who said they'll look into it.

The code that was deleted essentially consisted of two parts:

(1) code that runs the test vectors against Java providers and Javascript libraries. I'm currently extending this part to other languages for the new format. E.g., rust, more javascript libraries and python.

(2) tests that do not use test vectors. These tests can of course run independent of Wycheproof. Maybe there are some more appropriate places for them. I have a few more ideas for additional tests, but haven't had the time yet to work on them.

[alext-w]

(2) tests that do not use test vectors. These tests can of course run independent of Wycheproof. Maybe there are some more appropriate places for them. I have a few more ideas for additional tests, but haven't had the time yet to work on them.

Maybe the Paranoid lib, as we touched upon in previous similar discussions in this repo?

[bleichenbacher-daniel]

(2) tests that do not use test vectors. These tests can of course run independent of Wycheproof. Maybe there are some more appropriate places for them. I have a few more ideas for additional tests, but haven't had the time yet to work on them.

Maybe the Paranoid lib, as we touched upon in previous similar discussions in this repo?

I'm not sure yet. My main focus is still on extending the test vectors and running them against a number of libraries. One issue is often to determine all the algorithm parameters and formats used by a specific implementation.

Wycheproof is a better fit for most of the additional tests. Paranoid works best if one has access to large collections of public keys and signatures and of course the necessary computational power to run the tests. I don't have that at the moment.
I'm still trying to get in touch with Google with the goal to get a more stable and productive environment so that the tests can be revived. But I'm also open to other options.

[J08nY]

To me, something like CryptoFuzz is a better fit. Originally by Guido Vranken who has taken it private. Now it seems Mozilla Security team is keeping it alive.

The advantage is that there are unified bindings to exercise a large amount of primitives in a large amount of implementations.

We have that for ECC only in ECTester.

[bleichenbacher-daniel]

Thanks for pointing out CryptoFuzz. That looks indeed quite nice. Unified interfaces would be great. I would already be happy if there were some common interfaces per language. Java looks good in this regard, rust has some widely used traits for some primitives but not for others. Javascript is a complete mess and this is unfortunately also the language where additional tests have the highest changes of finding vulnerabilities.
Currently, I'm writing code for timing and key generation. In this case, I can just generate some JSON files containing data that can be analyzed later.