Can't compile ZED anymore using Vendor - error: failed to run custom build command for `webrtc-sys v0.3.5

[AngryPenguinPL]

Check for existing issues

• Completed

Describe the bug / provide steps to reproduce it

The last version of ZED that I managed to compile using the vendored rust crates was 0.164.2. Later versions give an error on compilation that says: failed to run custom build command for `webrtc-sys v0.3.5

Here is the error on Zed version 0.170.0-pre

error: failed to run custom build command for `webrtc-sys v0.3.5 (https://github.com/zed-industries/livekit-rust-sdks?rev=060964da10574cd9bf06463a53bf6e0769c5c45e#060964da)`
DEBUG util.py:461:  Caused by:
DEBUG util.py:461:    process didn't exit successfully: `/builddir/build/BUILD/zed-0.170.0_pre-build/zed-0.170.0-pre/target/release/build/webrtc-sys-fa68416f9d313caf/build-script-build` (exit status: 101)
DEBUG util.py:461:    --- stdout
DEBUG util.py:461:    cargo:rerun-if-env-changed=LK_DEBUG_WEBRTC
DEBUG util.py:461:    cargo:rerun-if-env-changed=LK_CUSTOM_WEBRTC
DEBUG util.py:461:    cargo:CXXBRIDGE_PREFIX=webrtc-sys
DEBUG util.py:461:    cargo:CXXBRIDGE_DIR0=/builddir/build/BUILD/zed-0.170.0_pre-build/zed-0.170.0-pre/target/release/build/webrtc-sys-a4b48bd95e534577/out/cxxbridge/include
DEBUG util.py:461:    cargo:CXXBRIDGE_DIR1=/builddir/build/BUILD/zed-0.170.0_pre-build/zed-0.170.0-pre/target/release/build/webrtc-sys-a4b48bd95e534577/out/cxxbridge/crate
DEBUG util.py:461:    --- stderr
DEBUG util.py:461:    CXX include path:
DEBUG util.py:461:      /builddir/build/BUILD/zed-0.170.0_pre-build/zed-0.170.0-pre/target/release/build/webrtc-sys-a4b48bd95e534577/out/cxxbridge/include
DEBUG util.py:461:      /builddir/build/BUILD/zed-0.170.0_pre-build/zed-0.170.0-pre/target/release/build/webrtc-sys-a4b48bd95e534577/out/cxxbridge/crate
DEBUG util.py:461:    thread 'main' panicked at /builddir/build/BUILD/zed-0.170.0_pre-build/zed-0.170.0-pre/vendor/webrtc-sys/build.rs:85:45:
DEBUG util.py:461:    called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("github.com")), port: None, path: "/zed-industries/webrtc/releases/download/m114_release_patched/webrtc-linux-x64-release.zip", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("dns error", Os { code: 16, kind: ResourceBusy, message: "Device or resource busy" })) }
DEBUG util.py:461:    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
DEBUG util.py:461:  warning: build failed, waiting for other jobs to finish...

Full build log: https://file-store.openmandriva.org/api/v1/file_stores/772278ed0f9367f6cf2c32e5396f4c7d272f3b49.log?show=true

I don't know what is visible in the livekit-rust-sdks log, but it is most likely not automatically added to the list of vendored rust dependencies.
In the vendor directory after executing "cargo vendor" I see only livekit, livekit-api, livekit-protocol and livekit-runtime, but there is no livekit-rust-sdk there...

The said webrtc-sys was vendored correctly.

I will add that I do not fully understand this error.

Does anyone know how to solve this?

Also looks like similar problem stopped opensuse package to build: https://build.opensuse.org/package/live_build_log/editors/zed/15.6/x86_64

Zed Version and System Specs

Zed from 0.166.0 to 0.170-pre

If applicable, add screenshots or screencasts of the incorrect state / behavior

No response

If applicable, attach your Zed.log file to this issue.

Zed.log

[notpeter]

1. Can you provide info about the platform you are trying to compile on (distro/version)?
2. Can you provide explicit steps/commands you are using to build (e.g. git clone, script/linux, cargo build --release, etc)
3. Looking at your log, there appears to be a DNS error resolving github.com are you trying to run this build somewhere offline or with restricted access ot the internet?

DEBUG util.py:461: called Result::unwrap() on an Err value: reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("github.com")), port: None, path: "/zed-industries/webrtc/releases/download/m114_release_patched/webrtc-linux-x64-release.zip", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("dns error", Os { code: 16, kind: ResourceBusy, message: "Device or resource busy" })) }

[AngryPenguinPL]

Os: OpenMandriva Cooker, but same problem is also on OpenSuse Tumbleweed.

1. Download released source code : https://github.com/zed-industries/zed/archive/refs/tags/v0.169.2.tar.gz
2. Unpack
3. Run inside source dir command to vendor rust dependencies: cargo vendor
4. Compress newly created vendor folder to tar.xz
5. Configure cargo to use vendored libs as:
   %cargo_prep -v vendor
6. Build with

# Build CLI
cd crates/cli/
cargo build --release
# Build Editor
cd ../zed/
cargo build --release --verbose

And yes, trying to build it outside of internet (unfortunately this is the policy of many linux systems, where the build must be performed without access to the network, using only libraries from the repository or vendored rust crates).
From this it follows that cargo is trying to download the missing file: zed-industries/webrtc/releases/download/m114_release_patched/webrtc-linux-x64-release.zip",
but it not have access to the network.
It seems to me that this file should be downloaded by "cargo vendor" and then vendored, but for some reason this does not happen.

[notpeter]

See: https://github.com/zed-industries/livekit-rust-sdks/blob/060964da10574cd9bf06463a53bf6e0769c5c45e/webrtc-sys/build/src/lib.rs#L98-L112
CC: @mgsloan

[mgsloan]

I believe that livekit's rust sdk is this way because it uses a patched libwebrtc. So it downloads a statically linked build of it to avoid needing to require all the build deps / build time of libwebrtc.

I wrote some docs after figuring out how to build with a patched version of libwebrtc, should be exactly what you need to build offline / fully from source: https://github.com/livekit/rust-sdks/blob/main/webrtc-sys/libwebrtc/README.md

It does seem pretty non-ideal to have a download of a precompiled binary as part of the build process. I'm not sure what a clean solution would look like (I don't yet know how vendoring C via crates works)

I recommend you use Zed's patched version. For some inscrutable reason, for me port numbers were getting serialized to gibberish, and this patch fixed the issue?!?! :

So Zed's fork of libwebrtc includes both LiveKit's patches and this patch.

[marv7000]

I'm the maintainer of Zed for openSUSE Tumbleweed.

As described by this bug report, Zed hasn't been building for a while with this exact error message because it isn't allowed to access the internet, which is a requirement to build in our build system.

Whatever webrtc-sys tries to download isn't caught by cargo vendor and seems like faulty behavior IMO.

[mgsloan]

@marv7000 I agree it's not good. One way to make progress on this might be to ask LiveKit to address this usecase - https://github.com/livekit/rust-sdks/

[hakimzulkufli]

@marv7000 @AngryPenguinPL I have found workaround for this. The build can be completed fully offline. Thanks to @mgsloan for writing that docs. That is the key to fix the build failing when offline.

Thankfully, Livekit developers added a custom directory support as a fallback when building webrtc-sys. The gist is to include the SDKs in the build package itself.

Add LK_CUSTOM_WEBRTC environment variable in zed/.cargo/config.toml to force webrtc-sys to use the pre-downloaded files. Then, determine the arch to use (Zed's patched for x64, Livekit's for arm64) and copy the files to the path set in the environment variable.

We can do this without touching Zed or the vendor files by adding the environment variable during build.

I made a branch of the package here for reference: https://build.opensuse.org/package/show/home:hzu/zed

[marv7000]

Neat, if you want you can create a submit request and I'll merge it.

[FalconSlayer51]

This is the problem with windows as well, but it can be fixed by installing latest Windows 11 SDK

[AngryPenguinPL]

I don't know why this bug was closed and the conversation moved to discussion.

The bug is still there and prevents Zed from being compiled on Linux distributions that care about security and can't use network access for compilation.

[hakimzulkufli]

I have managed to build it fully offline, completely from source, including WebRTC.

The catch? You need a ton of space and ton of time. Building Zed fully offline from source requires you to build WebRTC, which requires the whole of Chromium build toolchain.

1. Instructions on how to provide your own WebRTC binary here:
   Can't compile ZED anymore using Vendor - error: failed to run custom build command for `webrtc-sys v0.3.5 #24512 (comment)

2. See my repository for openSUSE Tumbleweed here for reference to build everything from source:
   https://build.opensuse.org/package/show/home:hzu/zed

If you don't mind using binaries, then you can simply preload the WebRTC binaries that was built by Zed and/or LiveKit and follow (1) only.

However, I still couldn't get it to work with ARM64. I don't have an ARM64 device to test and I don't want to waste openSUSE's resources for this. The error is caused by Google's depot_tools. It could be because I packaged the Chromium build toolchain that is only suitable for x86_64. I'm not sure as I don't have many experience with the toolchain.