Clarity on how {renv} uses the site-library

[mmuurr]

Apologies if (i) this has been discussed already or (ii) if there's a setting somewhere that help manage this (but I've looked and can't readily find any).

To describe my confusion, let's use a hypothetical machine on which:

• R and R's core libraries are installed at /usr/local/lib/R/library.
• Some additional site-wide packages have been installed at /usr/local/lib/R/site-library.

I believe that setup closely aligns with quite a few installations, but in general any 'core-library' and 'site-library' set should work the same.

Let's also assume that {renv} has never been used on this machine. There is no {renv} cache or {renv}-managed package library yet.

Scenario 1

I'm given a {renv.lock} file from which I decide to renv::restore() the project. This lock file includes the {somepkg} v1.2.3 dependency. It just so happens that {somepkg} v1.2.3 is installed in the site-library. When restore() is running, two things happen (amongst others):

1. The core-library is linked via {renv}'s sandbox.
2. {somepkg} v1.2.3 is downloaded, compiled, and installed.

In step 2, the site-library version of {somepkg} v1.2.3 was ignored, and {renv} favored fetching a fresh copy from CRAN (or whatever repo) instead. I suppose that makes sense in an effort to maximize consistency and isolation, but ...

Scenario 2

Before being given the above renv.lock file, I do some other development and happen to create a project that depends on {somepkg} v1.2.3, which again happens to be installed in the site-library. When I renv::init() the project, {renv} doesn't download {somepkg} v1.2.3, but rather it copies from the site-library.

Later, I'm given that renv.lock file from above. When I now restore() that project, {renv} decides to use the cached version of {somepkg} v1.2.3, which was copied from the site-library on account of the other project I created above.

That is to say, by virtue of Scenario 2 simply init()ing a (completely unrelated) project, we now have an alternative outcome in the two scenarios, where they have different 'sources' of {somepkg} v1.2.3 installed into {renv}'s cache.

Question

Since Scenario 2 is actually quite common (at least for me and amongst my colleagues), I don't quite understand why in Scenario 1 {renv} doesn't copy from site-library during the restore(). My confusion has some practicality: I have a few Docker workflows and there are a handful of patterns from which to choose (some of which -- but not all -- are similar to those described in the {renv} docs on Docker).

In one particular use case, I base some images off of the rocker/tidyverse family, but find when I then inject the {renv} bootstrapping files I still need to install much of the {tidyverse} (at least during the first layer build). In cases where the rocker/tidyverse image already contains exactly the correct package versions (installed in the site-library), I'd much rather just copy those to the {renv} cache. I've developed a workaround to achieve this (basically just a forced variant of Scenario 2), but workarounds don't feel great and yet I can't really find a compelling reason why this one wouldn't work, since on a local/development/interactive machine, Scenario 2 is indeed the most common anyhow.

Again, apologies if this isn't the correct venue for clarity, though I think it'd be useful during a restore if there was an option to say, "I trust the existing site-library, let's just use that as if we had run an init() on some other project anyhow".

[kevinushey]

If I understand correctly, in Scenario 1, renv should be copying the package from the site library -- this comes from here:

renv/R/retrieve.R

Line 301 in 8cb0525

renv_retrieve_libpaths

and so is controlled by the 'renv.config.install.shortcuts' option, but that should be TRUE by default. My best guess is that one or more of the checks in this function are failing:

renv/R/retrieve.R

Lines 611 to 644 in 8cb0525

	renv_retrieve_libpaths_impl <- function(record, libpath) {
	# form path to installed package's DESCRIPTION
	path <- file.path(libpath, record$Package)
	if (!file.exists(path))
	return(FALSE)
	# read DESCRIPTION
	desc <- renv_description_read(path = path)
	# check if it's compatible with the requested record
	fields <- c("Package", "Version", grep("^Remote", names(record), value = TRUE))
	compatible <- identical(record[fields], desc[fields])
	if (!compatible)
	return(FALSE)
	# check that it was built for a compatible version of R
	built <- desc[["Built"]]
	if (is.null(built))
	return(FALSE)
	ok <- catch(renv_description_built_version(desc))
	if (!identical(ok, TRUE))
	return(FALSE)
	# check that this package has a known source
	source <- renv_snapshot_description_source(desc)
	if (identical(source$Source, "unknown"))
	return(FALSE)
	# OK: copy this package as-is
	renv_retrieve_successful(record, path)
	}

You might try re-running scenario 1 with

debug(renv:::renv_retrieve_libpaths_impl)

and see why the version of your package in the site-library is being skipped. Perhaps the check for Remote fields in the DESCRIPTION file is too aggressive? Or perhaps something else is tripping renv up?

[kevinushey]

@mmuurr: were you able to try the debugging steps I suggested above and get any more information?

[mmuurr]

@kevinushey I got sidetracked, but will spin up a reprex with Docker this coming week and follow-up.