Fallback md5 is used, when trying hard to only use OpenSSL

[xnox]

Bug report

Bug description:

When

• OpenSSL is configured in FIPS mode
• recommended config is used to only load "base + fips" providers
• without the default provider
• CPython is compiled with --with-builtin-hashlib-hashes=blake2 to exclude fallback implementation of MD5

upon importing hashlib fails to create MD5 construct.

# python3.10 -c 'import hashlib'
ERROR:root:code for hash md5 was not found.
Traceback (most recent call last):
  File "/usr/lib/python3.10/hashlib.py", line 137, in __get_openssl_constructor
    f(usedforsecurity=False)
ValueError: [digital envelope routines] unsupported

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/hashlib.py", line 261, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/lib/python3.10/hashlib.py", line 141, in __get_openssl_constructor
    return __get_builtin_constructor(name)
  File "/usr/lib/python3.10/hashlib.py", line 123, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type md5

Reference implementation is upstream openssl 3.3.0, with enable-fips, fipsinstall completed and openssl.cnf set to

# cat /etc/ssl/openssl.cnf 
config_diagnostics = 1
openssl_conf = openssl_init

.include /etc/ssl/fipsmodule.cnf

[openssl_init]
providers = provider_sect
alg_section = algorithm_sect

[provider_sect]
fips = fips_sect
base = base_sect

[base_sect]
activate = 1

[algorithm_sect]
default_properties = fips=yes

In essence, things work well only when "default + fips" providers are loaded, as then MD5 functions in OpenSSL are detected as available and are used at runtime and correctly get blocked.

When only "base + fips" providers are loaded, ValueError is raised by OpenSSL constructor, and instead fallback implementation used from _md5 module if it was compiled in.

It seems like the above configuration was not tested, however it can be made to work. CPython should try to load the "default" OpenSSL provider, to guarantee access to non-fips hashes.

Security concerns

This is FedRAMP/FIPS compliance by-pass. This issue may allow using md5 without specifying "usedforsecurity=False" on systems otherwise configured to be in FIPS-mode only. And is the primary reason why documentation mentions that certain distributors of python remove md5 module altogether.

CPython versions tested on:

3.10, 3.11, 3.12

Operating systems tested on:

Linux

Linked PRs

• gh-118224: Load default OpenSSL provider for nonsecurity algorithms #118236
• [3.12] gh-118224: Load default OpenSSL provider for nonsecurity algorithms (GH-118236) #118238
• [3.11] gh-118224: Load default OpenSSL provider for nonsecurity algorithms (GH-118236) #118239
• [3.10] gh-118224: Load default OpenSSL provider for nonsecurity algorithms (GH-118236) #118240
• [3.9] gh-118224: Load default OpenSSL provider for nonsecurity algorithms (GH-118236) #118264

[encukou]

By --with-builtin-hashlib-hashes=blake2, you are explicitly saying that you do not want Python to provide md5. By setting base & fips in openssl.cnf, you're saying that you don't want OpenSSL to provide md5. Taken together, you've disabled md5.

I don't see why should Python should not honor those decisions. If you want md5, why don't you enable one of the things that provide it?

[xnox]

By --with-builtin-hashlib-hashes=blake2, you are explicitly saying that you do not want Python to provide md5. By setting base & fips in openssl.cnf, you're saying that you don't want OpenSSL to provide md5. Taken together, you've disabled md5.

I don't see why should Python should not honor those decisions. If you want md5, why don't you enable one of the things that provide it?

It is example of how to trigger the bug. The compliance bypass is that python silently fallsback to internal md5 (when compiled --with-builtin-hashlib-hashes=md5, aka default) when openssl one is incorrectly deemed unavailable (due to provider selection). When in fact, for non security purposes it is there and accessible (aka usage with indicator).

Note, openssl here is not compiled with no-md5.

[encukou]

If you believe that this is as security issue in python, please contact the security team -- see the security page (as linked from the Security policy links on GitHub).

But, I don't see the “compliance bypass”. CPython is not FIPS compliant; there isn't anything to bypass. There are some tools/options that make building a FIPS-certifiable CPython easier, but we're not really in a position to test that they do give you a compliant build.
Asking CPython to bypass system OpenSSL configuration, in order to support a build we don't really test in CI, is... a big thing to ask.

Note that building with --with-builtin-hashlib-hashes=blake2 could also be a compliance bypass. (Or are you planning to get that implementation FIPS-certified?)

cc @SethMichaelLarson

See also recent FIPS discussion.

[xnox]

If you believe that this is as security issue in python, please contact the security team -- see the security page (as linked from the Security policy links on GitHub).

But, I don't see the “compliance bypass”. CPython is not FIPS compliant; there isn't anything to bypass. There are some tools/options that make building a FIPS-certifiable CPython easier, but we're not really in a position to test that they do give you a compliant build. Asking CPython to bypass system OpenSSL configuration, in order to support a build we don't really test in CI, is... a big thing to ask.

The CI tests are already there for the FIPS mode. If desired I can add the extra matrix test case to address this issue as an explicit check.

I'm talking about honoring settings for the NIST algorithms, and using accelerated code-paths by default. And ensuring that existing functionality usedforsecurity=True/False api of hashlib.md5 actually works correctly as documented. Without changing any other behaviour of any other algorithms. When used with openssl as a cryptographic module, or when used with generic one.

Note that building with --with-builtin-hashlib-hashes=blake2 could also be a compliance bypass. (Or are you planning to get that implementation FIPS-certified?)

FIPS is consent based. Blocking usage of unapproved algorithms is relevant within a cryptographic module only. And those that want to certify cpython they can choose themselves if they want to block blake2 usage out right, or if they want to implement a runtime FIPS indicator w.r.t. usage of blake2 (such that apps that use blake2 are indicated as running in unapproved mode) which is a possibility under FIPS certification.

cc @SethMichaelLarson

See also recent FIPS discussion.

That discussion takes a narrow view of FIPS certification. Hashes that are not used for security purposes do not need to be FIPS certified (ie. siphash, xxhash64, and so on are perfectly fine to use for pyc hashing; and for key-value dictionary implementations). As long as said things are not used to protect data at rest (e.g. file encryption) / in transit (e.g. TLS) / used for authentication (e.g. HTTP basic-auth).

[xnox]

If you believe that this is as security issue in python

it's not a CVE in a classic sense of it. As compliance is frankly arbitrary and conflicting. (i.e. FIPS wants to use FIPS algorithms, and somebody else wants to use Cammelia, and someone else wants to use GOST).

My patch here, is scoped to ensure that MD5 (and in the future SHA1) are treated fairly and correctly: honoring system configuration, using accelerated (openssl) codepath, and have usedforsecurity=True/False operate as documented.

I'm not trying to exert any opinion of how one should configure the cpython builds; nor how one should configure their openssl; or how one should configure their FIPS modules. And ensure that usedforsecurity=False works correctly in more combinations.

[encukou]

I'll need to leave this to security experts. Hopefully the discussion is useful.

One thing I don't get: why don't you load “default” in the OpenSSL config?
AFAIK, md5 works (with usedforsecurity=False only) when you use the “default+fips” providers rather than “base+fips”. I don't see a reason to use the latter, other than not wanting to use “default”.

[xnox]

I'll need to leave this to security experts. Hopefully the discussion is useful.

One thing I don't get: why don't you load “default” in the OpenSSL config? AFAIK, md5 works (with usedforsecurity=False only) when you use the “default+fips” providers rather than “base+fips”. I don't see a reason to use the latter, other than not wanting to use “default”.

Please read carefully https://github.com/openssl/openssl/blob/master/README-PROVIDERS.md and especially the note w.r.t. FIPS provider under base provider.

if no providers are loaded - a fallback one is loaded, in vanilla upstream it is the default;
if explicit providers are provided (i.e. null) only those are loaded, as per fips provider documentation it is recommended to use fips+base;
many fips distributors of openssl actually make base+fips the fallback, rather than default - such that without any config default provider is not loaded, but fips+base is instead; lots of people misconfigure this and needlessly load base+default or default+fips.

I find it sad that "base" provider is a subset of "default" provider, as that is very confusing duplication of code at runtime.

Loding default+fips provider instead of base+fips provider has 3.5MB runtime memory usage penalty in most common cases, due to loadinging/init/iterating all the algorithms as available; which are then later filtered out by the "fips=yes" query.

The cpython CI tests as implemented right now, also do not currently setup openssl.cnf in a way that is most comonly deployed by openssl-fips distributors. I.e. when compared with "canonical openssl fips" or "redhat openssl fips" or even "openssl project fips - as per recommended docs"

[encukou]

I've read the documentation, and I don't see any hint of languages (or frameworks/libraries) being expected to silently load the default provider when asked for md5.

From Python's point of view, the user has made two explicit choices:

• Python is configured to disable md5.
• OpenSSL is configured to disable md5.

I do not think it's Python's place to silently override these choices.

The “user” here might be a redistributor or OS author. I get that this sucks for the end user, but Python is not at fault for md5 not being available. You should complain to whoever disabled it.

Feel free to post on Discourse to seek a second opinion.

I would consider a PR that exposes OSSL_PROVIDER_load -- that is, allow the user to override system & build-time defaults with another explicit choice.

[xnox]

I've read the documentation, and I don't see any hint of languages (or frameworks/libraries) being expected to silently load the default provider when asked for md5.

From Python's point of view, the user has made two explicit choices:

• Python is configured to disable md5.
• OpenSSL is configured to disable md5.

I do not think it's Python's place to silently override these choices.

The “user” here might be a redistributor or OS author. I get that this sucks for the end user, but Python is not at fault for md5 not being available. You should complain to whoever disabled it.

Feel free to post on Discourse to seek a second opinion.

I would consider a PR that exposes OSSL_PROVIDER_load -- that is, allow the user to override system & build-time defaults with another explicit choice.

Ok fine.

But what about this case:

• openssl configured in fips mode without md5 using base+fips providers only
• python built with builtin md5 fallback
• python today, skips openssl codepath and uses builtin md5 with usedforsecurity=True

Surely, cpython should not fallback to builtin md5 when openssl is configured in fips mode without md5. And only do so when usedforsecurity=False?

I.e. I don't see cpython today checking what the default context query string is / default intention. It is checking if "md5" is resolved or not, but it is not checking why it was not resolved.

[encukou]

That sounds more reasonable, though it might be harder to implement.
IMO, it'd be best to document to build Python without md5, and require OpenSSL, if you plan to enable FIPS mode.

[xnox]

That sounds more reasonable, though it might be harder to implement. IMO, it'd be best to document to build Python without md5, and require OpenSSL, if you plan to enable FIPS mode.

True, that blocks using md5 for secure purposes (encryption / authentication / signing / kdf / etc). But will not enable using md5 for non-security purposes (e.g. identifying all court documents that are referenced by md5) as usedforsecurity=False will be broken, unless things are done the way I am proposing it.

Separately, I find it odd that "import hashlib" tries to import and initialize "hashlib.md5". I wonder if it is time to start to require "hashlib.md5" and "hashlib.sha1" as imports.... or at least lazy inits.

[gpshead]

Relevant discussion - https://discuss.python.org/t/python-and-openssl-fips-mode/51389

[xnox]

I do not think it's Python's place to silently override these choices.

is it expected for hashlib.md5 to be always available? why is it always loaded upon hashlib import? Can it be made a lazy import? The documentation has some strong words about shipping python without hashlib.md5.

[encukou]

That's a good question. I guess CPython can go two ways:

• Deprecate hashlib.algorithms_guaranteed, as it's definitely possible to build Python without some of the algorithms, or
• Add strong language to --with-builtin-hashlib-hashes, saying that if you break hashlib.algorithms_guaranteed or algorithms_available, you're on the hook for explaining it to your users.

[xnox]

you're on the hook for explaining it to your users

Let me try to make this prettier, as the current traceback is very confusing and doesn't explain what is happening (it looks like python internals are broken, rather than runtime expectations do not match the ones that were assumed at build time - i.e. w.r.t. availability of md5).

[shashankram]

For FIPS, it's essential that hashlib.md5 doesn't get initialized upon import hashlib. What's the guidance to get around this and prevent this import error?
How do you get around the fact that modules may be directly consuming md5? E.g.
https://github.com/encode/httpx/blob/7c0cda153d301bde9a011e1dd7157d7e2b20889d/httpx/_auth.py#L177

[encukou]

That's really up to the organization that seeks FIPS certification for the system that contains the module.

AFAIK, things like “FIPS modes” are just ways to make it easier to create and audit a system that meets FIPS requirements. It's hard to give guidance for a specific module without knowing the rest of the system.

Anyway, I doubt that hashlib.md5 can't get initialized: you can still use md5 with usedforsecurity=False (but you need to audit all such usage).

[xnox]

For FIPS, it's essential that hashlib.md5 doesn't get initialized upon import hashlib. What's the guidance to get around this and prevent this import error? How do you get around the fact that modules may be directly consuming md5? E.g. https://github.com/encode/httpx/blob/7c0cda153d301bde9a011e1dd7157d7e2b20889d/httpx/_auth.py#L177

This is going to be a shameless plug - each distributions choose their own way. In Chainguard Python FIPS Image , I chose to implement this runtime behavior:

$ docker run -ti --entrypoint python cgr.dev/chainguard-private/python-fips:latest-dev
Python 3.12.4 (tags/v3.12.4-dirty:8e8a4ba, Jun 26 2024, 12:07:40) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> hashlib.md5
<built-in function openssl_md5>
>>> hashlib.md5()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
_hashlib.UnsupportedDigestmodError: [digital envelope routines] unsupported
>>> hashlib.md5(usedforsecurity=False)
<md5 _hashlib.HASH object @ 0x7b3e6ba96930>
>>> 

[encukou]

That's pretty much what RHEL does too.

[xnox]

If that is the consensus among most Linux distributions shipping FIPS and python..... could we get this upstream into cpython? Because vendor specific behaviour, even when all of them agree in FIPS mode, is counter productive.

[encukou]

Chime in here: https://discuss.python.org/t/python-and-openssl-fips-mode/51389/7
Yes, I imagine that having more than one vendor work on this would make it more acceptable.
Currently it's hard to find the boundary between stuff all FIPS distros need and platform-specific quirks. (On top of the problems of a feature that benefits US government suppliers but is at least inconvenient for almost everyone else.)

[lVlayhem]

The solution might be to support a new compiler option to not fallback to a default MD5 implementation when passing hashlib.md5(usedforsecurity=True)

[stratakis]

@xnox if you can please join the discourse discussion, I'd love some more feedback there to potentially figure out a solution for the FIPS related issues if that's feasible.

[xnox]

Ok! I had more thoughts on how to do things, and also learning approaches other systems are doing. So I think better implementation can be done which is more intuitive.