Refactor ClientRequest #11012

Dreamsorcerer · 2025-05-25T13:05:57Z

Goals

Separate logic between the proxy version and the main version, which have different requirements. This reduces need for asserts etc. where we know an attribute will not be None in one case, but not the other. The proxy code now uses the internal ClientRequestBase, while ClientRequest is for the main request and will be exposed to middlewares.
Rename some attributes/methods to better reflect what middleware users should be accessing (and matching what we've actually documented).
Ensure body is always Payload (it was only not when None was passed previously), and strip out the redundant code that tries to send an empty body.
Make it easier to make additions to ClientRequest in future releases without breaking subclasses.

TODO

If we're happy with this generally, then I still need to fix up the tests, which will take a substantial amount of work.

for more information, see https://pre-commit.ci

aiohttp/client_reqrep.py

@@ -28,8 +29,9 @@
 from multidict import CIMultiDict, CIMultiDictProxy, MultiDict, MultiDictProxy
 from yarl import URL

-from . import hdrs, helpers, http, multipart, payload
+from . import hdrs, multipart, payload


aiohttp/client_reqrep.py

…into clientrequest-refactor

aiohttp/client_reqrep.py

Dreamsorcerer · 2025-05-25T13:22:38Z

aiohttp/client_reqrep.py

-    body: Any = b""
-    auth = None
-    response = None
+class ClientResponse(HeadersMixin):


There are 0 changes to ClientResponse, I've just moved it up so I can reference it in ClientRequest.

I was under the impression the design was to be able to GC ClientRequest but hold on to ClientResponse

I don't that makes a difference? I've just added a reference to ClientResponse (the class, not an instance) as a class attribute in ClientRequest, so need it defined first.

codecov · 2025-05-25T13:34:08Z

❌ 217 Tests Failed:

Tests completed	Failed	Passed	Skipped
3874	217	3657	40

View the top 3 failed test(s) by shortest run time

tests.test_client_request::test_connection_key_with_proxy

Stack Traces | 0.002s run time

#x1B[0m#x1B[94masync#x1B[39;49;00m #x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92mtest_connection_key_with_proxy#x1B[39;49;00m() -> #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
    #x1B[90m    #x1B[39;49;00m#x1B[33m"""Verify the proxy headers are included in the ConnectionKey when a proxy is used."""#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        proxy = URL(#x1B[33m"#x1B[39;49;00m#x1B[33mhttp://proxy.example.com#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
>       req = ClientRequest(#x1B[90m#x1B[39;49;00m
            #x1B[33m"#x1B[39;49;00m#x1B[33mGET#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
            URL(#x1B[33m"#x1B[39;49;00m#x1B[33mhttp://example.com#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m),#x1B[90m#x1B[39;49;00m
            proxy=proxy,#x1B[90m#x1B[39;49;00m
            proxy_headers={#x1B[33m"#x1B[39;49;00m#x1B[33mX-Proxy#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m: #x1B[33m"#x1B[39;49;00m#x1B[33mtrue#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m},#x1B[90m#x1B[39;49;00m
            loop=asyncio.get_running_loop(),#x1B[90m#x1B[39;49;00m
        )#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31mE       TypeError: ClientRequest.__init__() missing 18 required keyword-only arguments: 'params', 'headers', 'skip_auto_headers', 'data', 'cookies', 'auth', 'version', 'compress', 'chunked', 'expect100', 'response_class', 'proxy_auth', 'timer', 'session', 'ssl', 'traces', 'trust_env', and 'server_hostname'#x1B[0m

proxy      = URL('http://proxy.example.com')

#x1B[1m#x1B[31mtests/test_client_request.py#x1B[0m:1524: TypeError

tests.test_client_request::test_get_with_data[pyloop]

Stack Traces | 0.002s run time

loop = <_UnixSelectorEventLoop running=False closed=False debug=False>

    #x1B[0m#x1B[94masync#x1B[39;49;00m #x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92mtest_get_with_data#x1B[39;49;00m(loop: asyncio.AbstractEventLoop) -> #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
        #x1B[94mfor#x1B[39;49;00m meth #x1B[95min#x1B[39;49;00m ClientRequest.GET_METHODS:#x1B[90m#x1B[39;49;00m
>           req = ClientRequest(#x1B[90m#x1B[39;49;00m
                meth, URL(#x1B[33m"#x1B[39;49;00m#x1B[33mhttp://python.org/#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m), data={#x1B[33m"#x1B[39;49;00m#x1B[33mlife#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m: #x1B[33m"#x1B[39;49;00m#x1B[33m42#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m}, loop=loop#x1B[90m#x1B[39;49;00m
            )#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31mE           TypeError: ClientRequest.__init__() missing 19 required keyword-only arguments: 'params', 'headers', 'skip_auto_headers', 'cookies', 'auth', 'version', 'compress', 'chunked', 'expect100', 'response_class', 'proxy', 'proxy_auth', 'timer', 'session', 'ssl', 'proxy_headers', 'traces', 'trust_env', and 'server_hostname'#x1B[0m

loop       = <_UnixSelectorEventLoop running=False closed=False debug=False>
meth       = 'OPTIONS'

#x1B[1m#x1B[31mtests/test_client_request.py#x1B[0m:810: TypeError

tests.test_client_request::test_connection_key_without_proxy

Stack Traces | 0.003s run time

#x1B[0m#x1B[94masync#x1B[39;49;00m #x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92mtest_connection_key_without_proxy#x1B[39;49;00m() -> #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
    #x1B[90m    #x1B[39;49;00m#x1B[33m"""Verify the proxy headers are not included in the ConnectionKey when a proxy is used."""#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        #x1B[90m# If proxy is unspecified, proxy_headers should be ignored#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
>       req = ClientRequest(#x1B[90m#x1B[39;49;00m
            #x1B[33m"#x1B[39;49;00m#x1B[33mGET#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
            URL(#x1B[33m"#x1B[39;49;00m#x1B[33mhttp://example.com#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m),#x1B[90m#x1B[39;49;00m
            proxy_headers={#x1B[33m"#x1B[39;49;00m#x1B[33mX-Proxy#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m: #x1B[33m"#x1B[39;49;00m#x1B[33mtrue#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m},#x1B[90m#x1B[39;49;00m
            loop=asyncio.get_running_loop(),#x1B[90m#x1B[39;49;00m
        )#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31mE       TypeError: ClientRequest.__init__() missing 19 required keyword-only arguments: 'params', 'headers', 'skip_auto_headers', 'data', 'cookies', 'auth', 'version', 'compress', 'chunked', 'expect100', 'response_class', 'proxy', 'proxy_auth', 'timer', 'session', 'ssl', 'traces', 'trust_env', and 'server_hostname'#x1B[0m


#x1B[1m#x1B[31mtests/test_client_request.py#x1B[0m:1538: TypeError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

bdraco · 2025-05-25T13:34:51Z

aiohttp/client_reqrep.py

@@ -190,623 +193,352 @@ class ConnectionKey(NamedTuple):
    proxy_headers_hash: Optional[int]  # hash(CIMultiDict)


-class ClientRequest:


Maybe we can back-port a limited version of this to keep back-ports more manageable. Right now automatic back-porting fails far too often, and we aren't any closer to releasing 4.x so we are going to have to deal with that for a long time which makes the risk of back-port errors higher.

Maybe. Will have to move backportable parts to another PR and merge, then update this one.

codspeed-hq · 2025-05-25T14:08:59Z

CodSpeed Performance Report

Merging #11012 will degrade performances by 9.34%

_{Comparing clientrequest-refactor (0cdf184) with master (4824648)}

Summary

❌ 1 regressions
✅ 58 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	`BASE`	`HEAD`	Change
❌	`test_send_client_request_one_hundred[pyloop]`	2.5 ms	2.8 ms	-9.34%

for more information, see https://pre-commit.ci

tests/test_benchmarks_client_request.py

…into clientrequest-refactor

aiohttp/client_reqrep.py

…into clientrequest-refactor

bdraco · 2025-05-26T06:56:30Z

Sorry about the conflicts

Bumps [pip](https://github.com/pypa/pip) from 24.3.1 to 25.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p> <blockquote> <h1>25.0 (2025-01-26)</h1> <h2>Deprecations and Removals</h2> <ul> <li>Deprecate the <code>no-python-version-warning</code> flag as it has long done nothing since Python 2 support was removed in pip 21.0. (<code>[#13154](pypa/pip#13154) <https://github.com/pypa/pip/issues/13154></code>_)</li> </ul> <h2>Features</h2> <ul> <li>Prefer to display :pep:<code>639</code> <code>License-Expression</code> in <code>pip show</code> if metadata version is at least 2.4. (<code>[#13112](pypa/pip#13112) <https://github.com/pypa/pip/issues/13112></code>_)</li> <li>Support :pep:<code>639</code> <code>License-Expression</code> and <code>License-File</code> metadata fields in JSON output. <code>pip inspect</code> and <code>pip install --report</code> now emit <code>license_expression</code> and <code>license_file</code> fields in the <code>metadata</code> object, if the corresponding fields are present in the installed <code>METADATA</code> file. (<code>[#13134](pypa/pip#13134) <https://github.com/pypa/pip/issues/13134></code>_)</li> <li>Files in the network cache will inherit the read/write permissions of pip's cache directory (in addition to the current user retaining read/write access). This enables a single cache to be shared among multiple users. (<code>[aio-libs#11012](pypa/pip#11012) <https://github.com/pypa/pip/issues/11012></code>_)</li> <li>Return the size, along with the number, of files cleared on <code>pip cache purge</code> and <code>pip cache remove</code> (<code>[#12176](pypa/pip#12176) <https://github.com/pypa/pip/issues/12176></code>_)</li> <li>Cache <code>python-requires</code> checks while filtering potential installation candidates. (<code>[#13128](pypa/pip#13128) <https://github.com/pypa/pip/issues/13128></code>_)</li> <li>Optimize package collection by avoiding unnecessary URL parsing and other processing. (<code>[#13132](pypa/pip#13132) <https://github.com/pypa/pip/issues/13132></code>_)</li> </ul> <h2>Bug Fixes</h2> <ul> <li>Reorder the encoding detection when decoding a requirements file, relying on UTF-8 over the locale encoding by default, matching the documented behaviour. (<code>[#12771](pypa/pip#12771) <https://github.com/pypa/pip/issues/12771></code>_)</li> <li>The pip version self check is disabled on <code>EXTERNALLY-MANAGED</code> environments. (<code>[#11820](pypa/pip#11820) <https://github.com/pypa/pip/issues/11820></code>_)</li> <li>Fix a security bug allowing a specially crafted wheel to execute code during installation. (<code>[#13079](pypa/pip#13079) <https://github.com/pypa/pip/issues/13079></code>_)</li> <li>The inclusion of <code>packaging</code> 24.2 changes how pre-release specifiers with <code><</code> and <code>></code> behave. Including a pre-release version with these specifiers now implies accepting pre-releases (e.g., <code><2.0dev</code> can include <code>1.0rc1</code>). To avoid implying pre-releases, avoid specifying them (e.g., use <code><2.0</code>). The exception is <code>!=</code>, which never implies pre-releases. (<code>[#13163](pypa/pip#13163) <https://github.com/pypa/pip/issues/13163></code>_)</li> <li>The <code>--cert</code> and <code>--client-cert</code> command-line options are now respected while installing build dependencies. Consequently, the private <code>_PIP_STANDALONE_CERT</code> environment variable is no longer used. (<code>[aio-libs#5502](pypa/pip#5502) <https://github.com/pypa/pip/issues/5502></code>_)</li> <li>The <code>--proxy</code> command-line option is now respected while installing build dependencies. (<code>[aio-libs#6018](pypa/pip#6018) <https://github.com/pypa/pip/issues/6018></code>_)</li> </ul> <h2>Vendored Libraries</h2> <ul> <li>Upgrade CacheControl to 0.14.1</li> <li>Upgrade idna to 3.10</li> <li>Upgrade msgpack to 1.1.0</li> <li>Upgrade packaging to 24.2</li> </ul>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pypa/pip/commit/f47b5874299848c688336ae7c8d69534013fe2c6"><code>f47b587</code></a> Bump for release</li> <li><a href="https://github.com/pypa/pip/commit/74a7f3335338712af44be95241daf62e756f27ec"><code>74a7f33</code></a> Update AUTHORS.txt</li> <li><a href="https://github.com/pypa/pip/commit/a008888a5b123e8d5e4667bdd21e4b42f3fc034c"><code>a008888</code></a> Merge pull request <a href="https://redirect.github.com/pypa/pip/issues/13171">#13171</a> from pypa/dependabot/github_actions/github-actions-...</li> <li><a href="https://github.com/pypa/pip/commit/d265fb7427c3ba4dbd10e4874a0bebea2e59350e"><code>d265fb7</code></a> Merge pull request <a href="https://redirect.github.com/pypa/pip/issues/13174">#13174</a> from ichard26/changelog</li> <li><a href="https://github.com/pypa/pip/commit/d35384ef91cb372a5223a01f980e5deb84c8fde5"><code>d35384e</code></a> Copyedit news entries before 25.0</li> <li><a href="https://github.com/pypa/pip/commit/adc4f9951b51b6a06e405b8960dd0c5f030f0fb5"><code>adc4f99</code></a> Reorder requirements file decoding (<a href="https://redirect.github.com/pypa/pip/issues/12795">#12795</a>)</li> <li><a href="https://github.com/pypa/pip/commit/40c42149a51a63e8416c047d5ddc0da1694387ea"><code>40c4214</code></a> Bump pypa/gh-action-pypi-publish in the github-actions group</li> <li><a href="https://github.com/pypa/pip/commit/6b0fb904803fbb3ce7da63966b2759407b6cd9dc"><code>6b0fb90</code></a> Merge pull request <a href="https://redirect.github.com/pypa/pip/issues/13048">#13048</a> from sbidoul/trusted-publisher-sbi</li> <li><a href="https://github.com/pypa/pip/commit/c7fb1e13ec79b1b48481ac245144c2b368e64f7d"><code>c7fb1e1</code></a> Merge pull request <a href="https://redirect.github.com/pypa/pip/issues/13145">#13145</a> from befeleme/pip-show-pep639</li> <li><a href="https://github.com/pypa/pip/commit/41c807c5938d269703c6ff2644fb3b7dc88eda4e"><code>41c807c</code></a> Show License-Expression if present in package metadata</li> <li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/24.3.1...25.0">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=24.3.1&new-version=25.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Refactor ClientRequest

18564e9

Dreamsorcerer requested a review from asvetlov as a code owner May 25, 2025 13:05

Dreamsorcerer added the backport:skip Skip backport bot label May 25, 2025

[pre-commit.ci] auto fixes from pre-commit.com hooks

f4d733f

for more information, see https://pre-commit.ci

github-advanced-security bot found potential problems May 25, 2025

View reviewed changes

Dreamsorcerer added 2 commits May 25, 2025 14:11

Add kwargs

22d724a

Merge branch 'clientrequest-refactor' of github.com:aio-libs/aiohttp …

6fa8996

…into clientrequest-refactor

Dreamsorcerer commented May 25, 2025

View reviewed changes

aiohttp/client_reqrep.py Outdated Show resolved Hide resolved

Tweaks

8257048

Dreamsorcerer commented May 25, 2025

View reviewed changes

Dreamsorcerer added 2 commits May 25, 2025 14:26

Fix

7171e0a

Fix

9023653

bdraco reviewed May 25, 2025

View reviewed changes

Dreamsorcerer added 2 commits May 25, 2025 14:50

Fix benchmarks

6a1e81a

Fix benchmarks

ef47363

Dreamsorcerer and others added 2 commits May 25, 2025 15:56

Remove unneeded code

477d52d

[pre-commit.ci] auto fixes from pre-commit.com hooks

8fbac96

for more information, see https://pre-commit.ci

github-advanced-security bot found potential problems May 25, 2025

View reviewed changes

tests/test_benchmarks_client_request.py Fixed Show fixed Hide fixed

Dreamsorcerer added 3 commits May 25, 2025 16:06

Fix

cb5f104

Merge branch 'clientrequest-refactor' of github.com:aio-libs/aiohttp …

3820c39

…into clientrequest-refactor

Merge branch 'master' into clientrequest-refactor

2559a0b

Dreamsorcerer commented May 25, 2025

View reviewed changes

aiohttp/client_reqrep.py Show resolved Hide resolved

Dreamsorcerer added 3 commits May 25, 2025 17:32

Readd shortcut

19dfe3a

Merge branch 'clientrequest-refactor' of github.com:aio-libs/aiohttp …

580387b

…into clientrequest-refactor

Tweaks

0cdf184

Dreamsorcerer mentioned this pull request May 26, 2025

Move ClientResponse to top of file #11025

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Refactor ClientRequest #11012

Refactor ClientRequest #11012

Uh oh!

Dreamsorcerer commented May 25, 2025 •

edited

Loading

Uh oh!

Check notice

Copilot Autofix

Uh oh!

Uh oh!

Dreamsorcerer May 25, 2025

Uh oh!

bdraco May 25, 2025

Uh oh!

Dreamsorcerer May 25, 2025

Uh oh!

codecov bot commented May 25, 2025 •

edited

Loading

Uh oh!

bdraco May 25, 2025

Uh oh!

Dreamsorcerer May 25, 2025

Uh oh!

codspeed-hq bot commented May 25, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

bdraco commented May 26, 2025

Uh oh!

Uh oh!

		@@ -190,623 +193,352 @@ class ConnectionKey(NamedTuple):
		proxy_headers_hash: Optional[int] # hash(CIMultiDict)


		class ClientRequest:

Uh oh!

Refactor ClientRequest #11012

Are you sure you want to change the base?

Refactor ClientRequest #11012

Uh oh!

Conversation

Dreamsorcerer commented May 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Goals

TODO

Uh oh!

Check notice

Uh oh!

Copilot Autofix

Uh oh!

Uh oh!

Dreamsorcerer May 25, 2025

Choose a reason for hiding this comment

Uh oh!

bdraco May 25, 2025

Choose a reason for hiding this comment

Uh oh!

Dreamsorcerer May 25, 2025

Choose a reason for hiding this comment

Uh oh!

codecov bot commented May 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 217 Tests Failed:

Uh oh!

bdraco May 25, 2025

Choose a reason for hiding this comment

Uh oh!

Dreamsorcerer May 25, 2025

Choose a reason for hiding this comment

Uh oh!

codspeed-hq bot commented May 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging #11012 will degrade performances by 9.34%

Summary

Benchmarks breakdown

Uh oh!

Uh oh!

Uh oh!

bdraco commented May 26, 2025

Uh oh!

Uh oh!

Dreamsorcerer commented May 25, 2025 •

edited

Loading

codecov bot commented May 25, 2025 •

edited

Loading

codspeed-hq bot commented May 25, 2025 •

edited

Loading