feat: allow using build backends from source

[baszalmstra]

This PR enables using build backends from source specifications. E.g. you can build build-backends from source! We already had the machinery in place to do this but we just quite didnt have this enabled in the manifest yet. This PR now unlocks doing:

[package.build]
backend = { name = "pixi-build-cmake", path = "https://github.com/prefix-dev/pixi-build-backends", subdirectory = "crates/pixi-build-cmake" }
channels = ["https://prefix.dev/conda-forge"]

This doesnt work just quite yet because we currently have no pixi.tomls defined for any of our build backends. But that should be an easy fix!

TODO

• Before merging this I think we should have a working test in our build testsuite.

[remimimimimi]

Just couple of comments, otherwise awesome!

[wolfv]

Looking forward to test drive this with a few build backends that I have cooking.

[baszalmstra]

@nichmor @remimimimimi I wonder if there is a way we can test this without having to resort to a complex python based end-2-end test.. Maybe you have thoughts?

[remimimimimi]

@nichmor @remimimimimi I wonder if there is a way we can test this without having to resort to a complex python based end-2-end test.. Maybe you have thoughts?

Could write tests in rust using combination of assert_cmd and assert_fs.

[nichmor]

@nichmor @remimimimimi I wonder if there is a way we can test this without having to resort to a complex python based end-2-end test.. Maybe you have thoughts?

How I was thinking is that we can pin the backends from pixi-build-backends in some pixi.toml that we will store in tests/data. But it will still use python e2e tests. Why you don't like it?

[baszalmstra]

Because it is relatively slow, it needs to fetch repodata, fetch packages, install packages, and run external programs. That's why I introduced the in-memory backend, it allows us to completely remove that.

[nichmor]

Because it is relatively slow, it needs to fetch repodata, fetch packages, install packages, and run external programs. That's why I introduced the in-memory backend, it allows us to completely remove that.

Yeah true - do you think we could build them also from source?

[ruben-arts]

I think it would be good to validate that it also rebuilds, this seems to work locally.

Also is there any way we can make this faster? This is a 25 second test.

[baszalmstra]

That seems odd! Since its using the in-memory backend!

[nichmor]

• wdym by rebuilds? if source code change, does it automatically reinstall the backend? if this - it will be fixed in a follow up PR
• regarding the speed - strange. I will take a look