Fails to install/load correctly on tuolumne with flux run

[matthiasdiener]

Fresh emirge installations (or reinstalling packages on prior installations) fail to correctly make packages loadable that are installed both editable and with setuptools when run with flux run:

$ flux run python -O -c "import logpyle; print(dir(logpyle))"
Error processing line 2 of /p/lustre5/diener3/Work_tuo/einsta/miniforge3/envs/ceesd/lib/python3.12/site-packages/__editable__.dagrt-2021.1.pth:

  Traceback (most recent call last):
    File "<frozen site>", line 206, in addpackage
    File "<string>", line 1, in <module>
    File "/p/lustre5/diener3/Work_tuo/einsta/miniforge3/envs/ceesd/lib/python3.12/site-packages/__editable___dagrt_2021_1_finder.py", line 7, in <module>
      from pathlib import Path
    File "/p/lustre5/diener3/Work_tuo/einsta/miniforge3/envs/ceesd/lib/python3.12/pathlib.py", line 20, in <module>
      from urllib.parse import quote_from_bytes as urlquote_from_bytes
  SyntaxError: source code string cannot contain null bytes

Remainder of file ignored
Error processing line 1 of /p/lustre5/diener3/Work_tuo/einsta/miniforge3/envs/ceesd/lib/python3.12/site-packages/__editable__.leap-2021.1.pth:

  Traceback (most recent call last):
    File "<frozen site>", line 206, in addpackage
    File "<string>", line 1, in <module>
    File "/p/lustre5/diener3/Work_tuo/einsta/miniforge3/envs/ceesd/lib/python3.12/site-packages/__editable___leap_2021_1_finder.py", line 7, in <module>
      from pathlib import Path
    File "/p/lustre5/diener3/Work_tuo/einsta/miniforge3/envs/ceesd/lib/python3.12/pathlib.py", line 20, in <module>
      from urllib.parse import quote_from_bytes as urlquote_from_bytes
  SyntaxError: source code string cannot contain null bytes

Remainder of file ignored
Error processing line 1 of /p/lustre5/diener3/Work_tuo/einsta/miniforge3/envs/ceesd/lib/python3.12/site-packages/__editable__.logpyle-2025.0.pth:

  Traceback (most recent call last):
    File "<frozen site>", line 206, in addpackage
    File "<string>", line 1, in <module>
    File "/p/lustre5/diener3/Work_tuo/einsta/miniforge3/envs/ceesd/lib/python3.12/site-packages/__editable___logpyle_2025_0_finder.py", line 7, in <module>
      from pathlib import Path
    File "/p/lustre5/diener3/Work_tuo/einsta/miniforge3/envs/ceesd/lib/python3.12/pathlib.py", line 20, in <module>
      from urllib.parse import quote_from_bytes as urlquote_from_bytes
  SyntaxError: source code string cannot contain null bytes

Remainder of file ignored
Error processing line 1 of /p/lustre5/diener3/Work_tuo/einsta/miniforge3/envs/ceesd/lib/python3.12/site-packages/__editable__.mirgecom-2020.1.pth:

  Traceback (most recent call last):
    File "<frozen site>", line 206, in addpackage
    File "<string>", line 1, in <module>
    File "/p/lustre5/diener3/Work_tuo/einsta/miniforge3/envs/ceesd/lib/python3.12/site-packages/__editable___mirgecom_2020_1_finder.py", line 7, in <module>
      from pathlib import Path
    File "/p/lustre5/diener3/Work_tuo/einsta/miniforge3/envs/ceesd/lib/python3.12/pathlib.py", line 20, in <module>
      from urllib.parse import quote_from_bytes as urlquote_from_bytes
  SyntaxError: source code string cannot contain null bytes

Remainder of file ignored
Error processing line 1 of /p/lustre5/diener3/Work_tuo/einsta/miniforge3/envs/ceesd/lib/python3.12/site-packages/__editable__.pyrometheus-2020.1.pth:

  Traceback (most recent call last):
    File "<frozen site>", line 206, in addpackage
    File "<string>", line 1, in <module>
    File "/p/lustre5/diener3/Work_tuo/einsta/miniforge3/envs/ceesd/lib/python3.12/site-packages/__editable___pyrometheus_2020_1_finder.py", line 7, in <module>
      from pathlib import Path
    File "/p/lustre5/diener3/Work_tuo/einsta/miniforge3/envs/ceesd/lib/python3.12/pathlib.py", line 20, in <module>
      from urllib.parse import quote_from_bytes as urlquote_from_bytes
  SyntaxError: source code string cannot contain null bytes

Remainder of file ignored
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/p/lustre5/diener3/Work_tuo/einsta/logpyle/logpyle/__init__.py", line 66, in <module>
    import importlib.metadata
  File "/p/lustre5/diener3/Work_tuo/einsta/miniforge3/envs/ceesd/lib/python3.12/importlib/metadata/__init__.py", line 7, in <module>
    import pathlib
  File "/p/lustre5/diener3/Work_tuo/einsta/miniforge3/envs/ceesd/lib/python3.12/pathlib.py", line 20, in <module>
    from urllib.parse import quote_from_bytes as urlquote_from_bytes
SyntaxError: source code string cannot contain null bytes
flux-job: task(s) exited with exit code 1

Packages involved seem to be dagrt, leap, logpyle, mirgecom, pyrometheus.
As far as I can tell, the relevant *.pth files do not contain null bytes.
Installing older versions of setuptools (and reinstalling packages) does not seem to fix the issue. Installing non-editable, or editable with hatchling, seems to work.

Note that this only happens when using flux run, not when running python directly.

cc @anderson2981 @inducer

[anderson2981]

submitted a ticket with the LC computing folks, they suggested either of the two options,

flux run -o spindle.level=off python -O -c "import logpyle; print(dir(logpyle))"

or

export SPINDLE_FLUXOPT=off

both fix the issue.

[matthiasdiener]

Here is the email from John Gyllenhaal regarding this setting:

Howdy Tuolumne, El Cap, and CORAL2 users;

On Thursday 4/3/25, at 5:50am PT, the "first" less capable fastload version was renamed to "fastload1" and "fastload" was symlinked to "fastload2". This makes the latest, recommended fastload implementation available under the clearer "fastload" name (instead of requiring fastload2). In addition, all fastload scripts have also been tweaked to be more flexible in their placement (described below).

In addition, Spindle is now the default launch accelerator on El Cap (as it is being brought up on the SCF), matching Tuolumne's default behavior for the last several months. Spindle accelerates all job launches and mitigates file system load at job launch. Currently, our other CORAL2 systems ( RZADAMS, TENAYA, TIOGA, and RZVERNAL) do not use Spindle by default. While Spindle is mostly optional for smaller runs (generally safe to not use Spindle if less than 33 nodes and if not launching a large number of jobs in a short period of time), we currently have it turned on for all jobs on El Cap and Tuolumne to allow us to detect and fix issues. Please report any Spindle issues found!

Fastload is a less capable job launch accelerator that may be used instead of Spindle in order to see if Spindle is triggering a particular issue or while waiting for a Spindle update to fix an issue. Fastload has some unfixable issues (some of which can be solved by unsetting the DISPLAY env variable) and Spindle does a much better job of mitigating file system load. This is why Spindle is needed on our big systems.

For those that cannot currently use Spindle for some reason for scale runs on Tuolumne and El Cap, you can now use "fastload" to get the latest fastload implementation as an alternative to Spindle. If it turns out if the original, less aggressively optimized, first fastload version works better for you, it can now be assessed via the new "fastload1" name.

If neither fastload or Spindle works for your use case at scale, there is a third CORAL1-ported-option build_libcache that is much better than using nothing (reach out to John Gyllenhaal for usage details).

NOTE: It has recently become clear that multinode parallel code builds should currently NOT be run under the current Spindle (or probably fastload). Compilers, makefiles, configures, and build systems tend to be pathologically bad stress tests for launch accelerators. If system tools like mkdir or compilers coredump unexpectedly, the first thing you should verify is that they are not running under Spindle/fastload, and if they are, stop running them under Spindle/fastload.

To turn off Spindle without turning on fastload, set the env variable "SPINDLE_FLUXOPT=off". For bash users, you can put SPINDLE_FLUXOPT=off before your flux or srun commands:

// Turn off Spindle for parallel builds that use flux in bash
SPINDLE_FLUXOPT=off flux run -n 2 --exclusive ./a.out
SPINDLE_FLUXOPT=off flux --parent alloc --nodes=2 --time-limit=1800s

--

The latest tweaks of the functionality of all the fastload script versions were to set environment variables to disable the flux's automatic loading of Spindle and fastload by flux. This allows the fastload to be more flexible in its placement on command lines. You can now place fastload either before the executable OR before flux commands. All the invocation methods below now work in our testing but it is new functionality, please report any issues found:

// The three original methods for using fastload instead of Spindle, but now can use the fastload name (instead of fastload2) before the executable
flux run -n 2 --exclusive --nodes=2 fastload ./a.out
srun -n 2 fastload ./a.out
flux run -o fastload=on -n 2 --exclusive --nodes=2 ./a.out

// New options for placing fastload before flux run and srun to use fastload instead of Spindle
fastload flux run -n 2 --exclusive --nodes=2 ./a.out
fastload srun -n2 ./a.out

// New options for placing fastload before allocations to use fastload instead of Spindle
fastload flux --parent alloc --nodes=2 --time-limit=1800s
fastload salloc -N 2 -t 30

If you have any issues with Spindle or Fastload, or have questions, please email the LC Hotline (lc-hotline@llnl.gov) and cc John Gyllenhaal (gyllenhaal1@llnl.gov)

[inducer]

As of inducer/grudge#383, leap and dagrt should no longer be dependencies. And pyrometheus and mirgecom could/should probably just switch to hatch.