Put cogs in redbot.ext_cogs package + refactor CogManager

[Jackenmen]

Description of the changes

This is pretty much a rehash of #2446 updated to work with the current codebase. Here are the relevant parts of the description from that PR:

This kicked off with me realising our use of Loader.load_module() as a hacky way to import modules from random locations is deprecated. Looking closer, I realised there were a couple of smelly things here which could be cleaned up. Here's what's changed:

• Paths in CogManager are now inserted into redbot.ext_cogs.__path__. This means to import any external cog (from install path, cogmanager path, whatever), the import path would be redbot.ext_cogs.cog_to_import. It makes the rest of the logic really straightforward.
• Importing cog modules is now done with importlib.import_module - this is a high-level function which allows us to delegate all of the nitty gritty import logic to Python itself, and it should be much more reliable. We're also no longer dealing with ModuleSpec objects, we just go straight from cog name to module.
• We're now using importlib.reload to reload modules like a good boy.
• Red.load_extension can take either a module, or a straight name. This means loading a package is as simple as Red.load_extension(package_name), no more needing to know what method of CogManager to use.
• Some internal documentation was added in a module docstring of redbot.core.cog_manager.

The only major deviation from the code of the original PR is the reload logic - the logic proposed by Toby is meant to deep reload the package, but a simple loop over sys.modules is not enough to achieve this, and what ends up happening instead is the package ends up with a mix of objects (classes, functions, etc.) from both the old and new module. This can cause some actual issues, as evidenced by, e.g. trying to load the Audio cog.

Instead, the reload logic left is kept pretty much as is, except that it uses importlib.reload() rather than the internal importlib._bootstrap._exec(). importlib.reload() does use importlib._bootstrap._exec() under the hood, so the observed behaviour should be identical.

Fixes #5956

Have the changes in this PR been tested?

Yes

[Jackenmen]

For future reference, here's a script for finding cogs in the Index that are affected by the changes in this PR:

check_6693.py

import re
import textwrap
from pathlib import Path

total_repo_count = 0
total_cog_count = 0
repo_count = 0
cog_count = 0


for repo_path in Path("cache").iterdir():
    if not repo_path.is_dir():
        continue
    total_repo_count += 1
    repo_name, *_ = repo_path.name.rpartition("_")
    repo_matches = {}
    for cog_path in repo_path.iterdir():
        total_cog_count += 1
        if not cog_path.is_dir():
            continue
        cog_name = cog_path.name
        escaped_cog_name = re.escape(cog_name)
        pattern = re.compile(
            fr"from +{escaped_cog_name}(\.[a-zA-Z0-9_]+)? +import"
            fr"|import +{escaped_cog_name}(\.[a-zA-Z0-9_]+)?($| |#)"
        )
        cog_matches = {}
        for file in cog_path.rglob("*.py"):
            with file.open(encoding="utf-8") as fp:
                for match in pattern.finditer(fp.read()):
                    file_matches = cog_matches.setdefault(file.relative_to(cog_path), [])
                    file_matches.append(match)
        if cog_matches:
            cog_count += 1
            repo_matches[cog_name] = cog_matches

    if repo_matches:
        repo_count += 1
        print("Found matches in", repo_name)
        for cog_name, cog_matches in repo_matches.items():
            print(f'  - Cog "{cog_name}":')
            for file, file_matches in cog_matches.items():
                print(f'    - File "{file}":')
                for match in file_matches:
                    print(f"      {match.group()}")


print()
print(
    f"{repo_count} repos out of {total_repo_count}"
    f" ({repo_count / total_repo_count * 100:.2f}%) affected"
)
print(
    f"{cog_count} cogs out of {total_cog_count}"
    f" ({cog_count / total_cog_count * 100:.2f}%) affected"
)

The results I got are:

16 repos out of 68 (23.53%) affected
50 cogs out of 1579 (3.17%) affected