Error: cannot allocate vector of size 1.1 Mb when running in parallel for > ~2500-4000 reps

[LHMarshall]

> whichmodel.sim <- make.simulation(reps = 8000,
+                                   design = equal.cover,
+                                   population.description = minkepop,
+                                   detectability = detect,
+                                   ds.analysis = strat.specific.or.not)
> crash.test <- run.simulation(whichmodel.sim, run.parallel = TRUE)
  |++++++++++++++++++++++                            | 43% ~01h 14m 48s  
 Error: cannot allocate vector of size 1.1 Mb 
11.
unserialize(node$con) 
10.
recvData.SOCKnode(con) 
9.
recvData(con) 
8.
FUN(X[[i]], ...) 
7.
lapply(cl[1:jobs], recvResult) 
6.
staticClusterApply(cl, fun, length(x), argfun) 
5.
clusterApply(cl = cl, x = splitList(X, nchunks), fun = lapply, 
    FUN = fun, ...) 
4.
do.call(c, clusterApply(cl = cl, x = splitList(X, nchunks), fun = lapply, 
    FUN = fun, ...), quote = TRUE) 
3.
PAR_FUN(cl, X[Split[[i]]], FUN, ...) 
2.
pbapply::pblapply(X = as.list(1:simulation@reps), FUN = single.sim.loop, 
    simulation = simulation, save.data = save.data, load.data = load.data, 
    data.path = data.path, cl = myCluster, counter = FALSE) at run.simulation.R#96
1.
run.simulation(whichmodel.sim, run.parallel = TRUE)

[LHMarshall]

Sounds like Cal may know some things that can help with this!

[lenthomas]

Feel free to contact him for comments - his work on parallel computing is on GPUs, but nonetheless he may have some useful insights.

[LHMarshall]

Information on improving memory usage: http://adv-r.had.co.nz/memory.html#:~:text=Despite%20what%20you%20might%20have,memory%20to%20the%20operating%20system.