Removing pathconfig warnings, defaulting the python path to include the local copy of the standard library

[zig-robotics]

Thanks for making this available! I made a few tweaks that allow the produced binary to be used out of the box. This allows for this package to be used as part of the build system. It may be better to point the prefix paths to the standard library as then we don't need to suppress the pathconfig warnings, but that would require copying the directory into the expected python3.11 directory which isn't straight forward to do as part of the build as we can't use lazy paths as flags. I'm open to suggestions to improve this though. I use this to do code generation thats required for building the ROS project. Here's one of the few places this gets used. Inline python in your build feels pretty cursed but I'm impressed it was so easy to remove a system dependency on python.

https://github.com/zig-robotics/zigros/blob/v0.1.0/rcutils/build.zig#L48-L67

[andrewrk]

Hmmm, this would be OK if it were only running python at build time, but if your goal is to produce a python that works at runtime, it's desirable to set the pythonpath at runtime. I suppose that's already possible. Have you tried setting the respective environment variable instead of hard-coding it with this macro?

[zig-robotics]

Thanks for the reply! Like you mention this doesn't stop you from specifying additional python paths at run time, though it dose dirty the default paths a bit. Here are some examples:

Path with original build

$ PYTHONPATH=./Lib ./zig-out/bin/cpython-original -c "import sys
print(sys.path)"
['', '/home/.../zig/cpython/Lib', '/usr/local/lib/python311.zip', '/usr/local/lib/python3.11', '/usr/local/lib/python3.11/lib-dynload']

Path with the proposed build, first with no PYTHONPATH env passed, then with the local lib passed and then with the system lib:

$ ./zig-out/bin/cpython -c "import sys
print(sys.path)"
['', '/usr/local/lib/python311.zip', '/home/.../zig/cpython/Lib', '/usr/local/lib/python3.11', '/usr/local/lib/python3.11/lib-dynload']
$ PYTHONPATH=./Lib ./zig-out/bin/cpython -c "import sys
print(sys.path)"
['', '/home/.../zig/cpython/Lib', '/usr/local/lib/python311.zip', '/usr/local/lib/python3.11', '/usr/local/lib/python3.11/lib-dynload']
$ PYTHONPATH=/usr/lib/python3.11/ ./zig-out/bin/cpython -c "import sys
print(sys.path)"
['', '/usr/lib/python3.11', '/usr/local/lib/python311.zip', '/home/.../zig/cpython/Lib', '/usr/local/lib/python3.11', '/usr/local/lib/python3.11/lib-dynload']

The issue I ran into with trying to set the environment variable on the run step is that the Lib directory is only accessible through a LazyPath, and there's no way to set environment variables with a LazyPath. It is possible to resolve this particular lazy path during the configure phase since we know it's not a generated path in this case, but that felt like abusing the LazyPath a bit. This same LazyPath thing is true when hardcoding the macro as well, but I thought b.path().getPath(b) was a little more obvious that it was okay to do despite the "Intended for use during the make phase only" warning that getPath comes with. At the time I thought it made more sense to me to address this in the python package as that makes the cpython artifact directly usable without this implicit requirement on pulling in lib, but I can see your side that this is really only a requirement if using python in the build system. The following does work:

const std = @import("std");

pub fn build(b: *std.Build) void {
    const python_command =
        \\import sys
        \\print(sys.path)
    ;
    const python_dep = b.dependency("python", .{});
    
    var python_step = b.addRunArtifact(python_dep.artifact("cpython"));
    python_step.setEnvironmentVariable("PYTHONPATH", python_dep.path("Lib").getPath(b));
    python_step.addArgs(&.{ "-c", python_command });
    
    b.getInstallStep().dependOn(&python_step.step);
}

Additionally in other areas of my project I wrap the underlying python running in a zig program to more easily get around generated LazyPath issues (among other things). This lets you pass python paths as actual LazyPaths which is required for some python dependencies that the project generates itself.
https://github.com/zig-robotics/zigros/blob/v0.1.0/rosidl/src/adapter_generator.zig#L41

Would moving all the python running to a zig wrapper be a more idiomatic approach?

In either case, are you okay with disabling the pathconfig warnings? The warning shows up in the build output even though there's no real issue (this is true with the runtime use case as well unless there happens to be a python3.11 directory in /usr/local)

[andrewrk]

I don't mind disabling those warnings, but it would be good to try and see if there is a different approach first before patching. The more patches we have the more effort it takes to pull in upstream changes, and the more work it takes possible users to decide whether it's OK to use and trust this package or not.

Edit: I didn't read your whole message yet, will come back to it later