Problem using DavMail on Linux with FIPS mode enabled

[esabol]

In case you're not familiar with what FIPS is, here is Google's explanation:

FIPS, or Federal Information Processing Standards, is a set of computer security standards that ensure cryptographic tools are implemented correctly. In Linux, FIPS mode enforces these standards by only allowing FIPS 140-2 approved encryption algorithms.
Many (United States) federal agencies mandate FIPS because it deals with sensitive data. FIPS requires stronger encryption requirements for in-flight and data address data.

The most notable difference is that FIPS disallows usage of MD5. (SHA256 is recommended instead.)

With FIPS enabled on RHEL 8.10, the latest DavMail release errors out with the following message:

Connect exception: java.lang.RuntimeException Unable to configure SunPKCS11 provider Initialization failed CKR_USER_TYPE_INVALID
        at davmail.exchange.ExchangeSessionFactory.handleNetworkDown(ExchangeSessionFactory.java:347)
        at davmail.exchange.ExchangeSessionFactory.checkConfig(ExchangeSessionFactory.java:324)
        at davmail.smtp.SmtpConnection.run(SmtpConnection.java:65)

This is with davmail.ssl.pkcs11Library=/usr/lib64/pkcs11/p11-kit-trust.so in ~/.davmail.properties.

Adding -Dcom.redhat.fips=false to the JAVA_OPTS setting in the davmail startup script bypasses this and DavMail will work with that setting, but I'm not sure that's technically allowed and I could see that workaround being temporary and not be an option in future Linux and/or Java releases.

If I google "Unable to configure SunPKCS11 provider Initialization failed CKR_USER_TYPE_INVALID", I see some other Java projects running into this too. One comment I saw said this: "It tried to initialize the SunPKCS11 using the configure file. If the test failed during FIPS testing, that is expected, because in FIPS mode, SunPKCS11 will be initialized using the FIPS nss.fips.cfg. And it can't be initialized again..." nss.fips.cfg is located at /etc/java/java-1.8.0-openjdk/java-1.8.0-openjdk-1.8.0.432.b06-2.0.1.el8.x86_64/lib/security/nss.fips.cfg on my RHEL 8.10 system, and it contains the following:

name = NSS-FIPS
nssLibraryDirectory = /usr/lib64
nssSecmodDirectory = sql:/etc/pki/nssdb
nssDbMode = readOnly
nssModule = fips

attributes(*,CKO_SECRET_KEY,CKK_GENERIC_SECRET)={ CKA_SIGN=true }

Not really sure what that means, to be honest.

Any ideas?

[mguessan]

Reviewing older tickets, any update o this?

Setting davmail.ssl.pkcs11Library to your PKCS11 lib and davmail.ssl.pkcs11Config with the reference nss.fips.cfg configuration seem like a good idea.

Alternative may be to catch exception in DavGatewaySSLSocketFactory.createSSLContext and/or detect this special use case.

[esabol]

No, I have no updates on this. I was hoping you or someone else had some ideas.

Setting davmail.ssl.pkcs11Library to your PKCS11 lib and davmail.ssl.pkcs11Config with the reference nss.fips.cfg configuration seem like a good idea.

I'm not sure I understand. What do we set davmail.ssl.pkcs11Config to??

With FIPS mode disabled, we don't set it to anything, and we set davmail.ssl.pkcs11Library to /usr/lib64/pkcs11/p11-kit-trust.so.

Alternative may be to catch exception in DavGatewaySSLSocketFactory.createSSLContext and/or detect this special use case.

Hmmm....