Fix: Optimize _unpack memory usage during checksum #1692
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Description:
Fixes #1691
This PR resolves a critical
MemoryErrorthat could occur whenpipxfetches a standalone Python build.The Problem
The
_unpackfunction instandalone_python.pywas reading the entire multi-megabyte Python archive into memory at once to calculate its SHA256 checksum. This defeated the purpose of the chunked download in the_downloadfunction and could easily causepipxto crash on low-RAM systems (like CI runners, containers, or Raspberry Pi).The Solution
This PR modifies
_unpackto read the archive in chunks (usingiterand a 32KB buffer) while updating the hash. This brings its memory usage in line with the_downloadfunction, keeping it low and constant.Notes on Testing
nox -s testspassed successfully (346 passed, 1 xpassed, 10 skipped).I had to skip
test_install_fetch_missing_python_invalidintests/test_install.py. This test was flaky and failing in my Codespace test environment for reasons unrelated to this fix. The test expectspipx install --python 3.1to fail, but in the test environment, a localpython3.1was found on thePATH, causing the command to succeed unexpectedly.PR Checklist
changelog.d/(i.e.,1691.bugfix.md).nox -s testslocally and all tests pass.