`espup install` detects and re-uses existing gcc, but always re-installs rust toolchain when it fails to execute it

[codyps]

Bug description

When running espup install after previously having installed the same esp toolchain, espup install reuses (and does not re-download) GCC or LLVM (and prints warnings that it is doing this re-use), but always re-downloads the rust toolchain.

Log of a typical run of espup install when the esp-rs toolchain is already installed.

[nix-shell:~]$ espup install
[2023-09-08T15:56:59Z INFO ] 💽  Installing the Espressif Rust ecosystem
[2023-09-08T15:56:59Z INFO ] 💡  Querying GitHub API: 'https://api.github.com/repos/esp-rs/rust-build/releases/latest'
[2023-09-08T15:57:00Z INFO ] 💡  Querying GitHub API: 'https://api.github.com/repos/esp-rs/rust-build/releases'
[2023-09-08T15:57:01Z INFO ] 🔧  Checking Rust installation
[2023-09-08T15:57:01Z INFO ] 🔧  Installing RISC-V targets ('riscv32imc-unknown-none-elf' and 'riscv32imac-unknown-none-elf') for 'nightly' toolchain
[2023-09-08T15:57:01Z WARN ] ⚠️   Previous installation of GCC exists in: '/home/cody/.rustup/toolchains/esp/xtensa-esp32s2-elf/esp-12.2.0_20230208'. Reusing this installation.
[2023-09-08T15:57:01Z WARN ] ⚠️   Previous installation of GCC exists in: '/home/cody/.rustup/toolchains/esp/xtensa-esp32s3-elf/esp-12.2.0_20230208'. Reusing this installation.
[2023-09-08T15:57:01Z WARN ] ⚠️   Previous installation of GCC exists in: '/home/cody/.rustup/toolchains/esp/riscv32-esp-elf/esp-12.2.0_20230208'. Reusing this installation.
[2023-09-08T15:57:01Z WARN ] ⚠️   Previous installation of LLVM exists in: '/home/cody/.rustup/toolchains/esp/xtensa-esp32-elf-clang/esp-16.0.0-20230516'. Reusing this installation.
[2023-09-08T15:57:01Z WARN ] ⚠️   Previous installation of GCC exists in: '/home/cody/.rustup/toolchains/esp/xtensa-esp32-elf/esp-12.2.0_20230208'. Reusing this installation.
[2023-09-08T15:57:01Z INFO ] 🔧  Uninstalling Xtensa Rust toolchain
[2023-09-08T15:57:01Z INFO ] 🔧  Installing Xtensa Rust 1.72.0.0 toolchain
[2023-09-08T15:57:01Z INFO ] 📥  Downloading file '/home/cody/.rustup/toolchains/esp/.tmpQJCDaO/rust.tar.xz' from 'https://github.com/esp-rs/rust-build/releases/download/v1.72.0.0/rust-1.72.0.0-x86_64-unknown-linux-gnu.tar.xz'
[2023-09-08T15:57:05Z INFO ] 🔧  Uncompressing tar.xz file to '/home/cody/.rustup/toolchains/esp/.tmpQJCDaO'
[2023-09-08T15:57:16Z INFO ] 🔧  Installing 'rust' component for Xtensa Rust toolchain
[2023-09-08T15:57:22Z INFO ] 📥  Downloading file '/home/cody/.rustup/toolchains/esp/.tmpRqAMrB/rust-src.tar.xz' from 'https://github.com/esp-rs/rust-build/releases/download/v1.72.0.0/rust-src-1.72.0.0.tar.xz'
[2023-09-08T15:57:22Z INFO ] 🔧  Uncompressing tar.xz file to '/home/cody/.rustup/toolchains/esp/.tmpRqAMrB'
[2023-09-08T15:57:23Z INFO ] 🔧  Installing 'rust-src' component for Xtensa Rust toolchain
[2023-09-08T15:58:23Z INFO ] 🔧  Creating export file
[2023-09-08T15:58:23Z WARN ] 💡  Please, set up the environment variables by running: '. /home/cody/export-esp.sh'
[2023-09-08T15:58:23Z WARN ] ⚠️   This step must be done every time you open a new terminal.
[2023-09-08T15:58:23Z INFO ] ✅  Installation successfully completed!

To Reproduce

Steps to reproduce the behavior:

1. espup install at least once
2. espup install a second time
3. Note that GCC & llvm aren't re-downloaded and re-installed, but the rust toolchain is

Expected behavior

Re-use the existing rust toolchain if it's already the right version, making re-running espup install a quick operation.

Environment

• OS: nixos
• espup version: 0.5.1-dev 2ac988b

[SergioGasquez]

Hi! It should reuse the installation, just tried, and I can't reproduce your issue:

❯ espup install
[2023-09-12T09:15:43Z INFO ] 💽  Installing the Espressif Rust ecosystem
[2023-09-12T09:15:43Z INFO ] 💡  Querying GitHub API: 'https://api.github.com/repos/esp-rs/rust-build/releases/latest'
[2023-09-12T09:15:43Z INFO ] 💡  Querying GitHub API: 'https://api.github.com/repos/esp-rs/rust-build/releases'
[2023-09-12T09:15:44Z INFO ] 🔧  Checking Rust installation
[2023-09-12T09:15:44Z INFO ] 🔧  Installing RISC-V targets ('riscv32imc-unknown-none-elf' and 'riscv32imac-unknown-none-elf') for 'nightly' toolchain
[2023-09-12T09:15:44Z WARN ] ⚠️   Previous installation of GCC exists in: '/home/sergio/.rustup/toolchains/esp/xtensa-esp32s3-elf/esp-12.2.0_20230208'. Reusing this installation.
[2023-09-12T09:15:44Z WARN ] ⚠️   Previous installation of GCC exists in: '/home/sergio/.rustup/toolchains/esp/xtensa-esp32s2-elf/esp-12.2.0_20230208'. Reusing this installation.
[2023-09-12T09:15:44Z WARN ] ⚠️   Previous installation of GCC exists in: '/home/sergio/.rustup/toolchains/esp/xtensa-esp32-elf/esp-12.2.0_20230208'. Reusing this installation.
[2023-09-12T09:15:44Z WARN ] ⚠️   Previous installation of GCC exists in: '/home/sergio/.rustup/toolchains/esp/riscv32-esp-elf/esp-12.2.0_20230208'. Reusing this installation.
[2023-09-12T09:15:44Z WARN ] ⚠️   Previous installation of LLVM exists in: '/home/sergio/.rustup/toolchains/esp/xtensa-esp32-elf-clang/esp-16.0.0-20230516'. Reusing this installation.
[2023-09-12T09:15:44Z WARN ] ⚠️   Previous installation of Xtensa Rust 1.72.0.0 exists in: '/home/sergio/.rustup/toolchains/esp'. Reusing this installation.
[2023-09-12T09:15:45Z INFO ] 🔧  Creating export file
[2023-09-12T09:15:45Z WARN ] 💡  Please, set up the environment variables by running: '. /home/sergio/export-esp.sh'
[2023-09-12T09:15:45Z WARN ] ⚠️   This step must be done every time you open a new terminal.
[2023-09-12T09:15:45Z INFO ] ✅  Installation successfully completed!

I think what happened to you is that you had an older Xtensa Rust version, and when you ran the espup install it updated your Xtensa Rust to the latest version.

[SergioGasquez]

Hello @jmesmon, did you have a chance to verify the behavior?¿

[codyps]

Ya, I took another look, and here's what is happening:

• When espup goes to check the current toolchain, it tries to execute the binaries.
• In my test system, the binaries failed to execute because I don't have the dynamic linker they're expecting
• espup sees that execution failure but doesn't print any error for the user and reinstalls the rust toolchain
• repeat.

(the "doesn't have the expected dynamic linker" bit is because this is nixos, which doesn't use the same filesystem layout. They carry patches to fixup binaries rustup downloads. I suspect I'll need to do the same thing for espup as long as it ships executables with a dependency on a dynamic linker in a particular location)

[SergioGasquez]

Do you have any suggestion on how to improve this on espup side? I don't Use Nix/NixOs

[codyps]

Things that could help here:

1. print a warning/error message when execution of the compiler fails when trying to detect the version
2. provide a hook in espup to run something (script/executable/etc) to post-process a toolchain after it installs. Should be part of some config/env that can be set for all espup invocations. This would (potentially) let me have something hooked into espup to fixup the dynamic linker path without patching espup source.
3. ship static executables for the rustc/cargo/other toolchain binaries so that no post-processing of the toolchain is needed.

[SergioGasquez]

Thanks for the suggestions, just created #357 which adds a warn message when failing to detect the version.

3. ship static executables for the rustc/cargo/other toolchain binaries so that no post-processing of the toolchain is needed.

This should be done in https://github.com/esp-rs/rust-build maybe we can open an issue for this.

[LucaFulchir]

I'm trying to make it work on NixOS, too.

after running espup install I ran a custom script to fix the path of the interpreters and system libraries of all executables under ~/.rustup

espup install will not reinstall things twice.

$ rustup show     
Default host: x86_64-unknown-linux-gnu
rustup home:  /home/luker/.rustup

installed toolchains
--------------------

nightly-x86_64-unknown-linux-gnu (default)
esp

active toolchain
----------------

esp (overridden by '/home/luker/Work/personal/EMM/emm/rust-toolchain.toml')
rustc 1.72.1-nightly (edc37d22d 2023-09-15) (1.72.1.0)

$ rustc -vV  
rustc 1.72.1-nightly (edc37d22d 2023-09-15) (1.72.1.0)
binary: rustc
commit-hash: edc37d22db1cc1b112b91addeb1f79951c58e661
commit-date: 2023-09-15
host: x86_64-unknown-linux-gnu
release: 1.72.1-nightly
LLVM version: 16.0.4

However, running cargo build on the generic template (no-std or std, esp32-s3 or esp32-c3) always terminates with:

error: Package `test v0.0.0 (/home/luker/.rustup/toolchains/esp/lib/rustlib/src/rust/library/test)` does not have the feature `backtrace`

...which does not make much sense to me.
But apparently it means I was using cargo from nixos instead of the one installed by espup, because then I tried:

$ export PATH="/home/luker/.rustup/toolchains/esp/bin:$PATH"

And it starts compiling. After a bit I get

error: linker `xtensa-esp32s3-elf-gcc` not found

..which actually already is in my path:

$ which xtensa-esp32s3-elf-gcc     
/home/luker/.rustup/toolchains/esp/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/xtensa-esp32s3-elf-gcc

...and now I'm stuck