i2d_PublicKey() fails with -1 for DILITHIUM2 key while using OQS provider with OpenSSL 3.2.1

[vk18anubhav]

Discussed in https://github.com/orgs/open-quantum-safe/discussions/1968

^{Originally posted by vk18anubhav October 30, 2024}
Hi Team,

I am generating a DILITHIUM2 key in my code using OpenSSL 3.2.1 by loading OQS provider as following

EVP_PKEY *pkey = NULL;
EVP_PKEY_CTX *ctx = (EVP_PKEY_CTX *)NULL;

pkey = EVP_PKEY_new();

ctx = EVP_PKEY_CTX_new_from_name(libctx, "dilithium2", NULL);

EVP_PKEY_keygen_init(ctx);

EVP_PKEY_keygen(ctx, &pkey);

While converting the key to DER format using i2d_PublicKey() , it fails with -1 while i2d_PrivateKey() succeeds.

len = i2d_PublicKey(pkey, NULL); // First call itself fails

if(len < buffer->len)
buffer->len = i2d_PublicKey(pkey, buffer); // This also fails 

I tried debugging the issue by putting breakpoints on i2d_PublicKey(), i2d_PrivateKey() and sharing observations

int i2d_PublicKey(const EVP_PKEY *a, unsigned char **pp)
{
    if (evp_pkey_is_provided(a)) {
        static const struct type_and_structure_st output_info[] = {
            { "DER", "type-specific" },
            { "blob", NULL },    /* for EC */
            { NULL, }
        };
 
        return i2d_provided(a, EVP_PKEY_PUBLIC_KEY, output_info, pp);
    }

int i2d_PrivateKey(const EVP_PKEY *a, unsigned char **pp)
{
    if (evp_pkey_is_provided(a)) {
        static const struct type_and_structure_st output_info[] = {
            { "DER", "type-specific" },
            { "DER", "PrivateKeyInfo" },
            { NULL, }
        };
 
        return i2d_provided(a, EVP_PKEY_KEYPAIR, output_info, pp);
    }

• i2d_provided() calls OSSL_ENCODER_CTX_new_for_pkey(), OSSL_ENCODER_to_data() in loop for members of output_info
• OSSL_ENCODER_CTX_new_for_pkey () was using oqsprovider for getting libctx
• OSSL_ENCODER_to_data() returns doing nothing for output_type= DER, output_structure= "type-specific" for both private and public key.
• OSSL_ENCODER_to_data() succeeds for next output_type= DER, output_structure= “PrivateKeyInfo" while it fails for output_type= blob, output_structure= NULL for public key and returns with -1

Can you please review on how I can resolve this ?

[SWilson4]

I think this issue is better suited to oqs-provider—we don't implement any encoding logic in liboqs. Transferring it to that project.

[baentsch]

I am not seeing anything oqsprovider specific in this question: This should be asked in an OpenSSL forum (or maybe is an FAQ there). For validation, could you please use a non-PQ algorithm in your test code and see whether that works as you expect @vk18anubhav ?

[wbeck10]

The problem is missing encoder for type-specific key.
Using the OpenSSL API "i2d_PublicKey" with an created dilithium2 pkey will fail because a "der + type-specific" decoder will be queried in the OQSProvider but this encoder is not available.
I see in encode_key2any.c comment out type-specific encoder function like
key_to_type_specific_der_bio. It seems for some reason, this type-specific encoder was deactivated or work in progress?
It is important to be able to convert a pkey into a der hex string.

OpenSSL i2d_PublicKey has following code party:
int i2d_PublicKey(const EVP_PKEY *a, unsigned char *pp)
{
if (evp_pkey_is_provided(a)) {
static const struct type_and_structure_st output_info[] = {
{ "DER", "type-specific" },
{ "blob", NULL }, / for EC */
{ NULL, }
};

    return i2d_provided(a, EVP_PKEY_PUBLIC_KEY, output_info, pp);
}

...
}

[baentsch]

I see in encode_key2any.c comment out type-specific encoder function like
key_to_type_specific_der_bio.

Thanks for pointing this out, @wbeck10 . This completely escaped my attention and I wonder why the provider could be used for such a long time (this is 3 year old code) without anyone noticing. As I'm a bit pressed for time, tagging this issue asking the community for help (removing the comment, adding probably missing logic and tests).

[wbeck10p]

I might can help you out here.
I'm working on a commit to get this working.

[wbeck10]

Pull request created:
#596

[baentsch]

Becomes irrelevant with Dilithium being dropped from liboqs. Please re-open if you think otherwise, @vk18anubhav (updating the issue to what you want to achieve with which other algorithm).