How does aiohttp certify certificates? I can get pretty much any other module BUT aiohttp to connect to discord.com.

[fatboychummy]

Title, essentially. I made an issue about this that has some more details (I probably should have made it a discussion in the first place), but essentially I cannot connect to discord with aiohttp (and since discord.py uses aiohttp, I cannot create a bot that I've been trying to create).

I've been working on this issue for around three and a half days now, every search I've done has come up with either random Mac issues or Windows issues, which are not helpful as I am running on Ubuntu Server 20.04. And other results just haven't helped any at all.

I can get other libraries like requests to work, though.

I'm mainly wondering if there's somewhere I can add certificates to, or if I can force aiohttp to certify in the same way that requests does (since it works).

The absolute minimum code I have is the following:

import asyncio
import aiohttp

async def main():
  async with aiohttp.ClientSession() as session:
    async with session.get('https://discord.com') as response:
      print('Status: ', response.status)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

And it causes the following:

Stack trace

Traceback (most recent call last):
File "/home/fatbot/.local/lib/python3.9/site-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
  return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
File "/usr/lib/python3.9/asyncio/base_events.py", line 1081, in create_connection
  transport, protocol = await self._create_connection_transport(
File "/usr/lib/python3.9/asyncio/base_events.py", line 1111, in _create_connection_transport
  await waiter
File "/usr/lib/python3.9/asyncio/sslproto.py", line 528, in data_received
  ssldata, appdata = self._sslpipe.feed_ssldata(data)
File "/usr/lib/python3.9/asyncio/sslproto.py", line 188, in feed_ssldata
  self._sslobj.do_handshake()
File "/usr/lib/python3.9/ssl.py", line 944, in do_handshake
  self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
  return future.result()
File "<stdin>", line 3, in main
File "/home/fatbot/.local/lib/python3.9/site-packages/aiohttp/client.py", line 1141, in __aenter__
  self._resp = await self._coro
File "/home/fatbot/.local/lib/python3.9/site-packages/aiohttp/client.py", line 536, in _request
  conn = await self._connector.connect(
File "/home/fatbot/.local/lib/python3.9/site-packages/aiohttp/connector.py", line 540, in connect
  proto = await self._create_connection(req, traces, timeout)
File "/home/fatbot/.local/lib/python3.9/site-packages/aiohttp/connector.py", line 901, in _create_connection
  _, proto = await self._create_direct_connection(req, traces, timeout)
File "/home/fatbot/.local/lib/python3.9/site-packages/aiohttp/connector.py", line 1206, in _create_direct_connection
  raise last_exc
File "/home/fatbot/.local/lib/python3.9/site-packages/aiohttp/connector.py", line 1175, in _create_direct_connection
  transp, proto = await self._wrap_create_connection(
File "/home/fatbot/.local/lib/python3.9/site-packages/aiohttp/connector.py", line 982, in _wrap_create_connection
  raise ClientConnectorCertificateError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host discord.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')]

[Dreamsorcerer]

I already answered your question: #7287 (comment)

[Dreamsorcerer]

Out of curiosity, I just looked at the requests code and as suspected it has it's own code to handle loading of certificates, which defaults to getting the certifi certificates.

As mentioned, aiohttp does nothing extra here, you need to look at SSLContext and either figure out why Python's default SSLContext is not loading certificates, or else load the certificates in a SSLContext explicitly.

[fatboychummy]

Yeah, sorry about the duplicate posting. I saw the issue got marked closed so went to make a discussion here instead but didn't notice the extra comments on the issue. I'll be trying a couple things out in a few hours here and will update if things worked out.

[fatboychummy]

For those wondering what happened to this -- I just gave up and full reinstalled my Linux system (though honestly that was something I've been needing to do for a while anyways).

It works now, so it must have been some obscure configuration issue on my end.

[webknjaz]

it must have been some obscure configuration issue on my end

Distros usually manage trusted root CAs. They have to be updated periodically. They are often packaged as ca-certificates or a similar package name. Consult with your distro docs.
Alternatively, you can configure trust in the SSLContext object by using your own collection of trusted certificates in a directory. A project called certifi ships such certificates via PyPI.

[fatboychummy]

I'll note that I both updated everything I could in relation to ca-certificates as well as installed certify -- neither of these worked at the time.

[webknjaz]

I'll note that I both updated everything I could in relation to ca-certificates

It may be that the default paths that OpenSSL/CPython are set up to look into may differ from the distro's default location. And where you got your Python from also matters — it might be compiled with different directories in mind.

as well as installed certify -- neither of these worked at the time.

When you install certifi, it doesn't automatically work — usually, you still need to retrieve it's data directory and feed it to the program that uses TLS. Some apps/libs (I think pip, for example) can get it from the env vars or have them vendored (requests?), but most don't do magic guessing since it's dangerous and the trusted certificates should be generally maintained by the distros, from the security PoV.

[Dreamsorcerer]

We should probably add some examples to the docs though, how to use certifi, how to load a client cert and similar. The Python docs on SSLContext are not the easiest to figure all that out, so some examples of the most common use cases would probably be helpful.

[Dreamsorcerer]

#7334