Fallback host rid is broken on non-portable builds

[tmds]

Microsoft's portable builds perform a fall-back to the portable rid when the /etc/os-release/ rid is not known.

This doesn't work for non-portable builds because the information for the portable rid is missing in Microsoft.NETCore.App.deps.json.

For example, below is the full runtimes section of a Fedora 37 build. It does not include a section for linux-x64 (which is the fallback host rid).

  "runtimes": {
    "fedora.37-x64": [
      "fedora.37",
      "fedora-x64",
      "fedora",
      "linux-x64",
      "linux",
      "unix-x64",
      "unix",
      "any",
      "base"
    ]

For source-build, there should be an exact match with the rid found here and the /etc/os-release rid.
That may not be the case:

• Because the /etc/os-release rid is unexpected (like in System.PlatformNotSupportedException: System.IO.Ports is currently only supported on Windows. on Archlinux #79196).
• It also happens when a distro gets upgraded, and the new version does not (yet) include packages for that .NET version. Then the existing .NET build will not know the new rid (fedora.38-x64 is unknown).

In these two cases, the Microsoft build will continue to work, while the source-build builds start failing.

We should include this information in the non-portable builds so the fallback works.

cc @ViktorHofer @ericstj @am11 @omajid

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Microsoft's portable builds perform a fall-back to the portable rid when the /etc/os-release/ rid is not known.

This doesn't work for non-portable builds because the information for the portable rid is missing in Microsoft.NETCore.App.deps.json.

For example, below is the full runtimes section of a Fedora 37 build. It does not include a section for linux-x64 (which is the fallback host rid).

  "runtimes": {
    "fedora.37-x64": [
      "fedora.37",
      "fedora-x64",
      "fedora",
      "linux-x64",
      "linux",
      "unix-x64",
      "unix",
      "any",
      "base"
    ]

For source-build, there should be an exact match with the rid found here and the /etc/os-release rid.
That may not be the case:

• Because the /etc/os-release rid is unexpected (like in System.PlatformNotSupportedException: System.IO.Ports is currently only supported on Windows. on Archlinux #79196).
• It also happens when a distro gets upgraded, and the new version does not (yet) include packages for that .NET version. Then the existing .NET build will not know the new rid (fedora.38-x64 is unknown).

In these two cases, the Microsoft build will continue to work, while the source-build builds start failing.

We should include this information in the non-portable builds so the fallback works.

cc @ViktorHofer @ericstj @am11 @omajid

Author:	tmds
Assignees:	-
Labels:	area-Infrastructure-libraries, untriaged
Milestone:	-

[am11]

I think we should work towards breaking free of /etc/os-release dependency. It does not scale at all.

Some context: In .NET 7, work was put to remove hard dependency on procfs (/proc) and now we don't fail in runtime and SDK even if you unmount /proc. The reason being, even on linux, there is no guarantee that procfs is present or functioning everywhere despite it is highly recommended facility provided by the linux kernel (vs., for instance, on FreeBSD, /proc is optional and not present by default).

/etc/os-release, in comparison to /proc, is a fragile resource for an ecosystem to depend on. It is a wrong contract for any serious operation. It was invented as part of systemd which not all distros use as their init system; but regardless, most distros add /etc/os-release as part of their base installation. We are parsing it in number of places and failing the execution if we find something (which we think is) wrong. If the fragility of /etc/os-release doesn't concern us then even better; we should write our own (equally fragile) ~/.dotnet/distro-info.json file and depend on that instead of something out of /etc which we don't control.

[tmds]

I think we should work towards breaking free of /etc/os-release dependency. It does not scale at all.

The behavior described in this issue is exactly that: a non-portable build should be able to consume portable assets (per fallback RID) independent of what's in /etc/os-release.

[ViktorHofer]

It also happens when a distro gets upgraded, and the new version does not (yet) include packages for that .NET version. Then the existing .NET build will not know the new rid (fedora.38-x64 is unknown).

What we discussed in NuGet/Home#5862 (especially NuGet/Home#5862 (comment)) touches on the bigger picture and would solve the uber issue instead of band-aiding broken source build scenarios like this N+1 scenario. Unfortunately, I don't think anyone currently considers picking this up from the backlog.

For example, below is the full runtimes section of a Fedora 37 build. It does not include a section for linux-x64 (which is the fallback host rid).

As I don't know much about the host fallback, can you please elaborate or link to how that fallback works today? Does a linux-x64 built host automatically fallback to "linux-x64" but then fails to find the associated parent rids because that information is missing from the graph? In respect to this ask, which fallback RID would we choose when source building for "fedora.37-x64"?

Anyway, this sounds like we are again working around the lack of a more flexible RID selection algorithm which presumably could supports floating versions. That isn't a problem per-se but if this will continue to seriously impact source build partners, we might instead or additionally strongly advocate for fixing the above linked issue.

[tmds]

Yes, I've been part of this discussion.

Early on, I was advocating to make the rids 'semantical'.

After @richlander put out the Simplify RID Model design, I think this much simpler model may be enough.

With that model, non-portable rids are for distinguishing source-built assets from portable ones (cfr dotnet/source-build#2932 (comment)).

What's proposed here is not a band-aid or a workaround. It's in line with this simplified model.

As I don't know much about the host fallback

The fallback is what makes Microsoft's build pick linux-x64 assets when the /etc/os-release rid is unknown.
That's why #79196 doesn't affect Microsoft builds.
This issue is about making the non-portable builds fall back too.

[am11]

The behavior described in this issue is exactly that:

Sure. The only difference is how we get there. I was proposing that we give this control to distros rather than owning ourselves. Ideally, it can be done by removing all calls to open(/etc/os-release) in the system. Be it building dotnet from source, publishing user projects (SDK) or during the run-time of user applications. Instead, we use something like open(/path/to/dotnet/root/distro-info.json) a resource which we will generate as part of the build using existing mechanics (source-build-dotnet.sh -p:TargetRid=..) for each distro differently, that will be installed on end-user's system. This way, we can completely remove these ever-growing chains of conditions

runtime/src/native/corehost/hostmisc/pal.unix.cpp

Line 756 in f8daacf

#else

We can read our format with simple code without specializing stuff like which distro uses version, how many parts of versions should be read, what if that version wasn't there etc. That would make it much easier to handle and rationalize proper fallbacks. Needless to say, this should be implemented in a way that it does not compromise any existing functionality for non-portable/portable scenarios and set us free from /etc/os-release for good.

[tmds]

@am11 it's an interesting idea, you should write it up in an issue.

I'd like to scope this to bring the non-portable build behavior in-line with the portable build when /etc/os-release has an unknown rid.

[tmds]

Rather than changing Microsoft.NETCore.App.deps.json to include the fallback rid, we may want to change the the hostpolicy so the fallback rid matches the source-build target rid.

@ViktorHofer the code for the fallback is here:

runtime/src/native/corehost/hostpolicy/deps_format.cpp

Lines 126 to 136 in 58719ec

	// If the current RID is not present in the RID fallback graph, then the platform
	// is unknown to us. At this point, we will fallback to using the base RIDs and attempt
	// asset lookup using them.
	//
	// We do the same even when the RID is empty.
	if (currentRid.empty() || (rid_fallback_graph.count(currentRid) == 0))
	{
	currentRid = pal::get_current_os_fallback_rid() + pal::string_t(_X("-")) + get_current_arch_name();
	trace::info(_X("Falling back to base HostRID: %s"), currentRid.c_str());
	}

It falls back to the corresponding portable rid defined here:

runtime/src/native/corehost/hostmisc/pal.h

Lines 78 to 90 in 58719ec

	#if defined(TARGET_FREEBSD)
	#define FALLBACK_HOST_RID _X("freebsd")
	#elif defined(TARGET_ILLUMOS)
	#define FALLBACK_HOST_RID _X("illumos")
	#elif defined(TARGET_SUNOS)
	#define FALLBACK_HOST_RID _X("solaris")
	#elif defined(TARGET_LINUX_MUSL)
	#define FALLBACK_HOST_RID _X("linux-musl")
	#elif defined(TARGET_ANDROID)
	#define FALLBACK_HOST_RID _X("linux-bionic")
	#else
	#define FALLBACK_HOST_RID _X("linux")
	#endif

Information about this portable rid is not present in a source-built Microsoft.NETCore.App.deps.json. So the fallback is broken.

The fix is either to add the fallback rid to Microsoft.NETCore.App.deps.json.
Or to change the fallback rid to one that is present in Microsoft.NETCore.App.deps.json: the target rid.

[ViktorHofer]

Or to change the fallback rid to one that is present in Microsoft.NETCore.App.deps.json: the target rid.

I spent some time yesterday thinking about this issue and what you propose is exactly what I imagined would be the right solution. Instead of finding a portable RID, just make sure that the target RID is honored in all scenarios (direct match and host fallback).

cc @vitek-karas