[help] memory = "transient" possibly not being respected in a dynamic target

[kendonB]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I have a large dynamic target that errors out, complaining about not enough memory. I suspect something is going wrong with the system giving memory back when targets are done.

_targets.R contains:

tar_option_set(
  storage = "worker", retrieval = "worker", memory = "transient",
  garbage_collection = TRUE
)

No reprex yet - I'm looking for guidance on how I might construct that in this instance.

I first run it without crew; it goes fine for a while getting through around 20 chunks out of 1000, ending in the following error:

# First run:
...
✖ errored branch parcels_with_buildings_021de68ebede58dc
...
Last error message:
    Opened 1.141MB (1196610 bytes) file ok but could not memory map it. This is a 64bit process. There is probably not enough contiguous virtual memory available.

I then run it again (same code) and the one that errored completes very fast.

# second run
▶ dispatched branch parcels_with_buildings_021de68ebede58dc
● completed branch parcels_with_buildings_021de68ebede58dc [19.11 seconds]

Any ideas on how to create a reprex for this situation (suspected memory issue)?

I'm on Windows 11 and targets 1.6.0.9001 2024-04-09 [1] Github (ropensci/targets@fd96c4e).

[kendonB]

In this case the code is running a spatial operation using sf. I recall a few instances where I thought sf was stealing memory until the process was killed and others have noticed similar things. Does targets currently run each individual dynamic target in a separate process? If not, is there a setting to do that?

[wlandau]

Does targets currently run each individual dynamic target in a separate process?

Not typically. And in tar_make() without crew, all targets run in the same process.

If not, is there a setting to do that?

tasks_max = 1 in crew_controller_local() will do this. With settings like tasks_max and seconds_idle (which come from settings in mirai), crew implements a smooth continuum between fully transient single-tasks workers and fully persistent workers which stay running until the pipeline completes.

[kendonB]

Ran it again monitoring memory usage in the Task Manager. The paging file usage is clearly building over time and is good evidence that R is not giving up memory. Will also try running it via WSL and see if I still see the same pattern.

[kendonB]

Similar behaviour in WSL

[kendonB]

I have been getting good behaviour with:

  tar_option_set(
    controller = crew::crew_controller_local(workers = 1)
  )

Bad behaviour is running with tar_make(use_crew = FALSE).

[wlandau]

To create a reprex, it may help to replace the sf objects with base R objects of roughly equal size. If the memory issues go away, then the problem could be e.g. sf accumulating data in the global environment. If the same issues persist, it would help to see how simple/small you can make the pipeline and still reproduce the error.