"can't find crate" when building Rust proc macro with remote execution

[cbarrete]

When building some proc macros with remote execution, rustc fails with the following error:

error[E0463]: can't find crate for `pyo3_macros_backend`
 --> third_party/pyo3-macros-0.26.0.crate/src/lib.rs:7:5
  |
7 | use pyo3_macros_backend::{
  |     ^^^^^^^^^^^^^^^^^^^ can't find crate

Note that this is not an issue with most proc macros, nor with local builds.
It seems like the .rlib for a dependency of the proc macro is not made available to it, presumably due to a bug in how dependencies are managed, or how the target/exec platforms are transitioned.

I have created a minimal repro at https://github.com/cbarrete/pyo3_repro. It should "work" out of the box on non-exotic Linux systems.

This may or may not be related to #1198.

I have tried messing with the BUCK file generated by Reindeer, but that made no difference, so I doubt that Reindeer is at fault here. I can file an issue for it if necessary though.

[cormacrelf]

This has nothing to do with exec platforms. It's a common pitfall when using buck to compile rust.

This is a side effect of how the prelude's rust rules do split compiles. The pyo3_macros_backend crate has been compiled twice with slightly different output, once for a metadata-only build and once for the codegen build. This is how the error usually manifests itself and rustc should do a better job -- it can show much better output in this case when you slap an extern crate pyo3_macros_backend. If you reindeer vendor; reindeer buckify and slap that extern crate at the top of vendor/pyo3-macros-0.26.0/src/lib.rs, you'll see this instead:

Remote command returned non-zero exit code 1
Remote action digest: `91f6cbf3b6fe1f3de5c7abd86eed8cba04b8b0891799ef49636c368e2eb7afb5:142`
stdout:
stderr:
error[E0460]: found possibly newer version of crate `pyo3_build_config` which `pyo3_macros_backend` depends on
 --> pyo3_repro/vendor/pyo3-macros-0.26.0/src/lib.rs:6:1
  |
6 | extern crate pyo3_macros_backend;
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = note: perhaps that crate needs to be recompiled?
  = note: the following crate versions were found:
          crate `pyo3_build_config`: buck-out/v2/gen/pyo3_repro/c895e1fc6d4688b5/___pyo3-macros-0.26.0__/MPPL-depslink-0/libpyo3_build_config-46a3569e.rlib
          crate `pyo3_macros_backend`: /nativelink/work/533caaab-3b49-4fc6-83ef-f40dca94cd8f/work/buck-out/v2/gen/pyo3_repro/c895e1fc6d4688b5/__pyo3-macros-backend-0.26.0__/LPPL/libpyo3_macros_backend-4575f58b.rlib


error: aborting due to 1 previous error


For more information about this error, try `rustc --explain E0460`.

This typically occurs when using a proc macro crate that has nondeterministic output, e.g. code generated in a different order due to hashmap randomization, or randomly generated identifiers in the macro output. A full explanation with an attached fix I've done for a library in the past is here. You don't see this error with Cargo because Cargo never uses metadata-only builds for anything but other metadata-only builds i.e. cargo check, whereas buck's prelude uses them to compile other dependent crates' codegen versions (thus creating lower latency and higher parallelism). Eventually the .rlib with codegen is linked when compiling a rust_binary (or indeed a proc macro) and that's when any nondeterminism rears its head.

In this case, it is most likely due to the pyo3-build-config crate configuring everything differently or non-deterministically or non-repeatably (like including timestamps in the output). Given the error message and dependency structure, I think it's actually the pyo3_build_config crate that is non-deterministic.

My stance on this is that proc macro crates and build scripts should be deterministic and you're saving yourself really mysterious caching issues by having this hard-error. Projects tend to accept PRs for making their output deterministic quite readily.

[cormacrelf]

buck2 build :pyo3-build-config-0.26.0[expand] --out=- > /tmp/abc.rs
buck2 clean
buck2 build :pyo3-build-config-0.26.0[expand] --local-only --no-remote-cache --out=- > /tmp/def.rs
git diff --no-index /tmp/abc.rs /tmp/def.rs

diff --git a/tmp/abc.rs b/tmp/def.rs
index 6724882631..1929c797dd 100644
--- a/tmp/abc.rs
+++ b/tmp/def.rs
@@ -2309,7 +2309,7 @@ const CONFIG_FILE: &str = "";
 /// cross-compilation settings.
 #[doc(hidden)]
 const HOST_CONFIG: &str =
-    "implementation=CPython\nversion=3.11\nshared=true\nabi3=false\nlib_name=python3.11\nlib_dir=/nix/store/5mxcy0d9pxrwsmk58c3lglqbczhckxlz-python3-3.11.14/lib\nexecutable=/nix/store/5mxcy0d9pxrwsmk58c3lglqbczhckxlz-python3-3.11.14/bin/python\npointer_width=64\nbuild_flags=\npython_framework_prefix=\nsuppress_build_script_link_lines=false\n";
+    "implementation=CPython\nversion=3.12\nshared=true\nabi3=false\nlib_name=python3.12\nlib_dir=/nix/store/jd20rkmqmkfkcvk2wl2lmzz7acq4svlr-python3-3.12.12/lib\nexecutable=/nix/store/jd20rkmqmkfkcvk2wl2lmzz7acq4svlr-python3-3.12.12/bin/python\npointer_width=64\nbuild_flags=\npython_framework_prefix=\nsuppress_build_script_link_lines=false\n";
 /// Returns the path where PyO3's build.rs writes its cross compile configuration.
 ///
 /// The config file will be named `$OUT_DIR/<triple>/pyo3-build-config.txt`.
@@ -2320,7 +2320,7 @@ fn resolve_cross_compile_config_path() -> Option<PathBuf> {
     env::var_os("TARGET").map(|target|
             {
                 let mut path =
-                    PathBuf::from("/nativelink/work/4eef80ce-8fe8-4f36-8916-1b81925b6518/work/buck-out/v2/gen/pyo3_repro/ecf8614efebd21d1/__pyo3-build-config-0.26.0-build-script-run__/OUT_DIR");
+                    PathBuf::from("/path/to/local/copy/pyo3_repro/buck-out/v2/gen/pyo3_repro/ecf8614efebd21d1/__pyo3-build-config-0.26.0-build-script-run__/OUT_DIR");
                 path.push(Path::new(&target));
                 path.push("pyo3-build-config.txt");
                 path

You will obviously get slightly different output. Notes:

• I deleted the fixup that added $(exe toolchains//:cpython) as in my repo that is a toolchain rule and cannot be used for something else. I didn't run your repro as-is because my RBE image is a bit too custom to work.
• Different pythons may not be an issue in your setup if you are using this successfully. Or, they may also be a problem due to absolute paths into buck-out once again including the UUID like below. It's certainly possible to embed a buck-out path into a binary, but it must be relative from repo root like buck-out/blah/blah. The $(exe ) macros etc generate this kind, fortunately. I wouldn't be surprised if pyo3 is absolutifying these paths though.
• The macro-expansion includes the absolute path to the remote build working directory. Including the UUID. I think in my case this is an "action ID" for remote execution, so even two actions within the same build invocation (metadata vs codegen) will produce different output when running remotely. If local is working it's because the absolute path to buck-out is consistent between builds. When local fails later it's because it's reusing some cache from the remote build.

Basically try changing env!("OUT_DIR") to std::env::var("OUT_DIR") in the build script (i.e. this code that is "only to be called from a build script"). That'll get you closer.

A usage like include!(concat!(env!("OUT_DIR"), "/blah.rs")) is not problematic because the macro expands to the contents of a file inside the OUT_DIR, and not the path. But this case is problematic because the fully expanded macro output includes a hardcoded directory (hardcoded in a loose sense). This is a bug for Cargo as well since the OUT_DIR baked into the expanded code and hence the build script binary will be a stale path if you move your Cargo project around on disk.

[cbarrete]

Woah, thank you so much, this was it. I don't know if I'd have ever figured it out on my own.

It's a common pitfall when using buck to compile rust.

I was not aware at all, is this referenced somewhere? If not, I'll probably make a PR to add this to a troubleshooting section somewhere.

Basically try changing env!("OUT_DIR") to std::env::var("OUT_DIR") in the build script (i.e. this code that is "only to be called from a build script"). That'll get you closer.

Unfortunately this doesn't quite work, because this code gets called in other build scripts, which have OUT_DIR set to their own value, but I'll figure something out.

[cormacrelf]

There are no docs on it. The [expand] diffing technique is worth documenting in addition to the rest. Feel free to use the diagram from the rocket PR and any of my comments there or here or elsewhere.