Adding support for specific KEM algorithms from liboqs in third party openssl providers.

[venkatesh6911]

I have added support for mlkem768 in my own openssl provider by referencing oqsprovider. The openssl provider also has support for classic asymmetric algorithms such as RSA, ECDSA, ECDH etc.

OpenSSL speed for mlkem768 has been working fine without any issues.

I am facing with bad mac error when testing for a handshake.

Note: mlkem768 is enabled as tls group at both server and client.

OpenSSL version: v3.2.3

Server:
./openssl s_server -provider qatprovider -cert /home/vej/certs/prime256cert.pem -key /home/vej/certs/prime256key.pem -www -tls1_3 -groups mlkem768

SSL log:
SSL3 alert read:fatal:bad record mac
SSL_accept:error in error
ERROR
40471D5D007F0000:error:0A0003FC:SSL routines:ssl3_read_bytes:ssl/tls alert bad record mac:ssl/record/rec_layer_s3.c:909:SSL alert number 20

Client:
./openssl s_client -connect 127.0.0.1:4433 -groups mlkem768

SSL log:
SSL3 alert write:fatal:bad record mac
SSL_connect:error in error
40C7FBF3E47F0000:error:0A000119:SSL routines:tls_get_more_records:decryption failed or bad record mac:ssl/record/methods/tls_common.c:866:
40C7FBF3E47F0000:error:0A000139:SSL routines::record layer failure:ssl/record/rec_layer_s3.c:689:

Question:
What can be the possible reasons for this error ?
My intention is to have a single provider which has the capabilities for both PQC and classic algorithms as opposed to having two providers (default and oqsprovider).

[t-j-h]

I'm curious as to why you are doing this now that OpenSSL-3.5-dev has ML-KEM, ML-DSA and SLH-DSA checked into the main branch.

[venkatesh6911]

Hi @t-j-h , thanks for replying to my query. Our provider does not support versions greater than 3.2 yet.

[baentsch]

Can you please point to your code base in github so I can take a look such as to not speculate, @venkatesh6911 ?

[baentsch]

On second reading,

My intention is to have a single provider which has the capabilities for both PQC and classic algorithms as opposed to having two providers

allow me to share the concern that by not relying on high-quality crypto for both, classic as well as PQ crypto, as well as not using proven-good RNGs, such provider may well be a risk to its users.

[venkatesh6911]

I fully agree. Mine is a experimental PoC to showcase performance to the product management by enabling kyber (mlkem) on our own provider. This work will not be approved for production.

[venkatesh6911]

I figured out that part which is causing the "bad mac" error. I was suppose to be setting public key (OSSL_PKEY_PARAM_ENCODED_PUBLIC_KEY) received from params in OSSL_FUNC_KEYMGMT_SET_PARAMS function. If there is no set_params function defined in the dispatch table, it complains of "bad mac" error.