Allow HSMs which only support `CKM_RSA_PKCS_PSS` / `CKM_ECDSA` mechanisms (digest as input) (ESPTOOL-1188)

[ulrichb]

Is your feature request related to a problem?

Currently, espsecure sign-data --hsm uses the SHA256_RSA_PKCS_PSS or ECDSA_SHA256 PKCS#11 mechanisms for the signature generation calls (see get_mechanism()).

These mechanisms do two things: hash the input data and afterwards perform the signing operation on the digest.

We wanted to test espsecure sign-data with our (SignPath) own PKCS#11 CryptoProvider which currently doesn't support these PKCS#11 mechanisms, but CKM_RSA_PKCS_PSS / CKM_ECDSA (which take an pre-calculated hash as input).

As this may be also an issue with other CryptoProviders / HSMs, it's maybe better to switch to the more low-level CKM_RSA_PKCS_PSS / CKM_ECDSA mechanisms in espsecure and calculate the input hash before (in the Python part).

What do you think about this idea?

[github-actions]

Hi @ulrichb! Please be aware that (1) the following suggestions are generated by a bot and haven't been fact-checked by Espressif Systems, (2) burning eFuses and enabling security features are irreversible operations and can damage your ESP32 device. We hope that this message will help you until an Espressif Engineer looks at your issue.

---

Thank you for raising this feature request and providing detailed context.

Based on the available Espressif documentation, the current implementation of espsecure sign-data --hsm indeed uses PKCS#11 mechanisms that combine hashing and signing (such as CKM_SHA256_RSA_PKCS_PSS and CKM_ECDSA_SHA256). This matches your observation and is also reflected in the official Espressif blog, which states that the initial support allows generating signed applications as per Secure Boot V2 scheme with either RSA-PSS 3072 or ECDSA NISTP256 algorithms, but does not specify support for mechanisms that require a pre-calculated digest as input.

The documentation does not mention support for HSMs that only provide the lower-level CKM_RSA_PKCS_PSS or CKM_ECDSA mechanisms (which expect the digest as input, not the raw data). Your suggestion—to pre-calculate the hash in Python and then use these mechanisms for signing—would indeed improve compatibility with a wider range of HSMs and CryptoProviders, including those like SignPath.

This is a reasonable and practical enhancement, as it would allow espsecure to interoperate with HSMs that do not implement the combined hash-and-sign mechanisms. The Espressif documentation does not indicate any technical blockers to this approach, and it aligns with standard cryptographic practices (pre-hashing data before signing).

Summary:
Your proposal to add support for CKM_RSA_PKCS_PSS and CKM_ECDSA (digest-as-input) mechanisms in espsecure is not currently implemented, but it is a valid and useful feature request. It would enhance compatibility with more HSMs and CryptoProviders. There is no indication in the official documentation that this approach would conflict with Espressif’s secure boot requirements, and it is consistent with standard PKCS#11 usage patterns.

References:

• Espsecure HSM support and PKCS#11 integration
• Espressif blog: Secure Signing Using External HSM

If you need further technical details or want to discuss implementation specifics, it may be helpful to open a discussion or pull request on the Espressif esptool repository.