Installing packages in parallel with restore()

[KasperSkytte]

Hi!

I'm using renv a heck of a lot across different computers and projects, great package. But this also means running renv::restore() several times a day, and it usually takes half an hour or more to install all the packages from scratch. I am hoping to speed this up as much as possible. Would it be possible to implement a sort of parallel installation of packages similar to fx install.packages(..., Ncpus = 8) when using renv::restore()? Or perhaps to pass arguments on to install.packages(...) through renv::restore(...), in particular the Ncpus argument? But I'm unsure if this would work with the (very nice!) caching of packages with renv, I'll leave that to you.

Regards
Kasper

[kevinushey]

Thanks for the feature request! This is definitely something I want to implement in the future; just haven't been able to carve out time for this yet.

[jucor]

This would be a fantastic addition :) Thanks for considering it! How can we help?

[KasperSkytte]

This is still highly relevant to me. How much work do you think it would take? I'll gladly contribute if it's not too complicated.

[yrro]

Try this workaround in your .Rprofile:

Sys.setenv(MAKEFLAGS = sprintf("-j%d", parallel::detectCores()))

[KasperSkytte]

@yrro That would only be for anything involving make, usually compiled code, and not actually installing multiple packages in parallel, independently. It does speed things up a bit, but most other steps are still taking long

[alonme]

@kevinushey - Any news? Any way we can help?

[kevinushey]

The development version of renv now has experimental support for pak, which can be used to install packages in parallel. For example:

options(renv.config.pak.enabled = TRUE)
.libPaths(tempdir())
renv::install("tidyverse")

pak will take care of downloading + installing all the required packages in parallel. On my M1 macOS machine, this completes pretty quickly:

✓ 1 pkg + 98 deps: kept 5, added 94, dld 94 (55.90 MB) [1m 30.5s]      

Note that pak does not yet support installation of packages from the CRAN archives, which we need for restore -- hopefully that will come in a future release. Once we're more certain the integration is solid we'll consider turning this on by default.

[alonme]

Thanks @kevinushey ,

Linking the relevant issue in pak r-lib/pak#343