Skip to content

[PB] Cleanup reproduce flow #49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 18, 2025
Merged

Conversation

thesofakillers
Copy link
Collaborator

@thesofakillers thesofakillers commented Jul 18, 2025

Our reproduction execution logic was clunky and unwieldy to work with. This PR streamlines it with a bundle of quality-of-life improvements:

  • Add a warning when salvaging will be skipped due to timeout configurations (otherwise it would silently skip)
  • Improve logging throughout the code
  • Remove stale TODOs
  • Ensure explicit support for blobfile
  • Consolidate ReproductionMeta and ReproductionOutput into a single structure
  • Pass dataclass instances directly instead of converting to and from dictionaries
  • Simplify reproduce_on_computer by removing:
    • Installation steps that can be handled in the Dockerfile
    • Inline try/except blocks (this logic can be moved elsewhere)
  • Defer computer start until it’s actually needed
  • Move run_reproduce_on_computer to paperbench/scripts/run_reproduce.py so all core reproduce logic is in one place
  • Address an existing TODO and add a new TODO to integrate with the ComputerInterface API

@thesofakillers thesofakillers changed the title PB] Cleanup reproduce flow [PB] Cleanup reproduce flow Jul 18, 2025
@thesofakillers thesofakillers merged commit 6369d90 into main Jul 18, 2025
3 checks passed
@thesofakillers thesofakillers deleted the pb/cleanup-reproduce-flow branch July 18, 2025 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant