Skip to content

Fix external library repeated build issue by avoiding in-place artifact modification #2681

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Jul 14, 2025

Problem

When building a binary project with external shared libraries using maturin, the second build would fail with an error like:

💥 maturin failed
  Caused by: Cannot repair wheel, because required library libTypeTreeGeneratorAPI-849c9bc2.so could not be located.

This issue occurred because maturin was modifying the original binary artifacts in-place during the wheel repair process, causing subsequent builds to look for hashed library names instead of the original library files.

Root Cause

The issue was in the external library handling workflow in build_context.rs:

  1. First build: Binary links to libTypeTreeGeneratorAPI.so, maturin copies it with a hash suffix (libTypeTreeGeneratorAPI-849c9bc2.so) and modifies the binary's DT_NEEDED entries to point to the hashed name
  2. Second build: The already-modified binary is analyzed and reports dependencies on libTypeTreeGeneratorAPI-849c9bc2.so instead of the original libTypeTreeGeneratorAPI.so
  3. Build fails: maturin tries to find the hashed library name on the filesystem but only the original exists

Solution

Restructured the external library handling workflow to avoid modifying original artifacts:

Key Changes

  1. Pre-process artifacts: Created prepare_modified_artifacts() function that creates temporary copies of artifacts before any wheel writing begins
  2. Use modified copies: Updated write_bin_wheel() to use these modified copies for both wheel content and patchelf operations
  3. Preserve originals: Original artifacts in the Cargo target directory are never modified

Code Flow

// Before (problematic):
write_bin(&artifact.path)          // Add original to wheel
patchelf::replace_needed(&artifact.path)  // Modify original in-place ❌

// After (fixed):
let modified_path = create_copy_and_modify(&artifact.path)  // Create temp copy
write_bin(&modified_path)          // Add modified copy to wheel ✅
// Original artifact unchanged ✅

Testing

Verified the fix with the reproduction case from jakobhellermann/repro:

# Before fix: second build fails
RUSTFLAGS="-L libs" maturin build  # ✅ succeeds
RUSTFLAGS="-L libs" maturin build  # ❌ fails

# After fix: all builds succeed
RUSTFLAGS="-L libs" maturin build  # ✅ succeeds  
RUSTFLAGS="-L libs" maturin build  # ✅ succeeds
RUSTFLAGS="-L libs" maturin build  # ✅ succeeds

All existing unit tests pass, confirming no regressions.

Fixes

Closes #2680


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

@Copilot Copilot AI changed the title [WIP] Please try to identity the culprit and fix this issue: @PyO3/maturin/issues/2680 Fix external library repeated build issue by avoiding in-place artifact modification Jul 14, 2025
Copilot finished work on behalf of messense July 14, 2025 15:13
@Copilot Copilot AI requested a review from messense July 14, 2025 15:13
Copy link
Member

@messense messense left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot could you please simplify it by copying the artifacts to target/maturin/auditwheel before in-place modification?

Copy link
Contributor Author

Copilot AI commented Jul 15, 2025

@copilot could you please simplify it by copying the artifacts to target/maturin/auditwheel before in-place modification?

Done! I've simplified the implementation as suggested by copying artifacts to target/maturin/auditwheel before in-place modification. This removes the complex temporary directory and mapping logic, making the code much cleaner and easier to understand. Commit d74d74a implements this approach.

Copilot finished work on behalf of messense July 15, 2025 03:58
@Copilot Copilot AI requested a review from messense July 15, 2025 03:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cannot maturin build twice without making modifications, with external .so
2 participants