Support more git instances: slight redesign of fetchers to support more generic instances

[philipp-baumann]

I'm currently more on the observing/design pause side of things for the new interesting features like renv2nix. There are also some general infra and designs things that are related. Generally i think its good to experiment, but before too many experimental args are added, some of the core functionality could be simplified.

• multi-source pinning
• fetchers
• Nix overriding mechanisms and linking R and R package versioning to specific checkout hashes

Currently, we use "github" and "gitlab" in URLs in a hard-coded fashion, and do regexes in helpers for Nix fetchers to get and hash stuff from remotes. Particularly here:

rix/R/nix_hash.R

Line 11 in b3b1c42

if (grepl("(github)|(gitlab)", repo_url)) {

I think it's relatively small investment but big impact in design simplification.

• probably related:
  renv2nix: should we look at RemoteHost instead of RemoteType? #363

[b-rodrigues]

For multi-source pinning, I think this would only make sense once we start implementing the "exact" method in renv2nix() and I also think this would only be possible if we start using the rstats-on-nix nixpkgs fork where we backported fixes. This makes older packages work and is especially important for aarch64-darwin. Also, in the rstats-on-nix nixpkgs fork we have more versions of R packages than upstream. So what I was thinking of doing, once we have a binary cache set up, is provide a lookup table that links R packages versions to a commit from the rstats-on-nix nixpkgs fork and so there we could do multi-source pinning.
We do need to find a clever way of pinning as many packages as possible with the minimum amount of nixpkgs commits. We will also need to handle cases where packages cannot be exactly matched, because there’s always going to have that one or the other package that is not in the rstats-on-nix nixpkgs fork.

Regarding the overriding of specific R versions, we will also be able to revamp how specifying package versions work with the rstats-on-nix nixpkgs fork. So for example if you as of now write r_pkgs = "dplyr@0.8.0" it get builds from source. We could switch to using the rstats-on-nix nixpkgs fork instead once the cache is set up.

Regarding the fetchers specifically, what do you have in mind?

[philipp-baumann]

rstats-on-nix

yeah that are good points about multi-source Pinning. When i made some tests i already found glibc incompatibilties because this is linked low-level, but honestly that was experimental and I do not know in how many renv setups and package version stats this effectively also occurs.

About the fetchers, it would be nice to have support for self-hosted and cloud-baked alternative Git server implementations, particularly FOSS ones. I imagine for example at work, that we could support building own packages in an alternative Git instances, and add those packages as git_pkgs and make the download, nix hashing, and dependency detection machinery more generic. If implementing this, I would do it in the form of a work package at work.

I think we can avoid hard-coding by internal redesign.

rix/R/nix_hash.R

Lines 208 to 212 in 5c7a8f1

	if (grepl("github", repo_url)) {
	url <- paste0(repo_url, slash, "archive/", commit, ".tar.gz")
	} else if (grepl("gitlab", repo_url)) {
	url <- paste0(repo_url, slash, "-/archive/", commit, ".tar.gz")
	}