Editable installs + multiprocessing break vector generation

[danceratopz]

After #4627 moved the project to uv (editable installs by default), our multiprocessing-based generators (pathos + dill) fail due to the editable installs. Normal Makefile development targets run fine.

Here's the docs for editable installs. Tldr: Editable installs are standard when working on a project in dev mode.

Reproduce the Issue

On b867e7d (fail in ~< 1m30s on laptop; success in ~6m30):

make reftests fork=phase0,altair preset=minimal threads=8

Root cause

For process-based pools (pathos.multiprocessing.ProcessingPool), pathos uses dill to pickle callables:

• By-reference (“module + qualname”) if they look cleanly importable.
• By-value if callables appear non-stable/non-importable, i.e. in editable intalls.
  - This triggers a walk of large/global object graphs (caches, C-extensions) → recursion and “cannot locate reference” warnings (e.g., _lru.LRU / dict-lru types; this is a C-extension).

Threads (pathos.pools.ThreadPool) don’t pickle functions and aren’t affected.

This only impacts the multiprocessing path. pytest-xdist is unaffected because it schedules by nodeid and each worker re-imports test modules locally (no pickling of callables).

Solution

1. Short-term: Keep uv run --no-editable for generator targets to stabilize releases:

UV_RUN_NE = uv run --no-editable

2. Medium-term: Refactor generators into pytest tasks/parametrized tests and parallelize with xdist to unify dev/release behavior and avoid surprises. And avoid maintaining a custom test collection and generation framework.

[leolara]

@danceratopz Pathos/dill was needed because of the ad-hoc test framework stores tests as lambdas that need to be passed to the test runners. Once, we move everything to pytest, this will not be a problem.

[jtraglia]

Something here is causing me troubles too. I'm modifying gen_runner.py but the changes are not taking effect when re-executing. Pretty sure this has something to do with editable installs. Even deactivate and make clean don't make a difference.