memory efficient parallel compting

[ArthurPERE]

Hello,

I have a question about the memory of the parallel. I am on linux so the parameter for the parallelism is MulticoreParam.

Each time a session spawn, the whole memory is copied to this session (at least that what I understand). But I don't want that the whole memory of the session to be copied. Is there a trick to be used to not do that, for exemple spawning a whole new session (even if it means putting aside a little performance), detached from the inital session

Thank you for your answer

[ArthurPERE]

And is there an outbreak for the problem describe here :
https://support.bioconductor.org/p/70196/#70509

I tried to write the function with gc() at the end, but to no avail, there is too much data in the session after the loop is finished.

[vjcitn]

See the note on copy-on-change.

SharedObject may also be relevant.

Apropos the support ticket you reference, please try to be more specific. Also indicate the result of sessionInfo() and whether you are working within Rstudio or in some other IDE.

[ArthurPERE]

I am working in Rstudio, but I tried it inline with Rscript, and my computer was strugling in term of memory (with RAM 96 Go, and 48 CPUs). And it was like in Rstudio.

And this is my code :

  Results.1.all =
    BiocParallel::bplapply(split(all_expermentation, seq(nrow(all_expermentation))),
      function(value, raw_counts, sub_set){
        
        experiment = NULL
        try(experiment <- coseq::coseq(
          raw_counts, K = value$k, model = "Normal", normFactors="TMM", transformation="arcsin",
          subset=sub_set, seed=6798+value$k+value$a-1))
        gc()
        return(experiment)
      }, raw_counts = Raw_Counts, sub_set = subset, BPOPTIONS = BiocParallel::bpoptions(packages = "coseq")
    )

And this is my sessionInfo() :

sessionInfo()
R version 4.4.3 (2025-02-28)
Platform: x86_64-pc-linux-gnu
Running under: Debian GNU/Linux 12 (bookworm)

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.11.0 
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.11.0

locale:
 [1] LC_CTYPE=fr_FR.UTF-8       LC_NUMERIC=C               LC_TIME=fr_FR.UTF-8        LC_COLLATE=fr_FR.UTF-8     LC_MONETARY=fr_FR.UTF-8    LC_MESSAGES=fr_FR.UTF-8   
 [7] LC_PAPER=fr_FR.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=fr_FR.UTF-8 LC_IDENTIFICATION=C       

time zone: Europe/Paris
tzcode source: system (glibc)

attached base packages:
[1] parallel  stats4    stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] devtools_2.4.5              usethis_3.1.0               ggplotify_0.1.2             pdftools_3.5.0              UpSetR_1.4.0                data.table_1.16.4          
 [7] coseq_1.30.0                SummarizedExperiment_1.34.0 Biobase_2.64.0              GenomicRanges_1.58.0        GenomeInfoDb_1.40.1         IRanges_2.38.1             
[13] S4Vectors_0.42.1            BiocGenerics_0.50.0         MatrixGenerics_1.16.0       matrixStats_1.5.0           RColorBrewer_1.1-3          dplyr_1.1.4                
[19] plyr_1.8.9                  ggpubr_0.6.0                reshape2_1.4.4              gplots_3.2.0                FactoMineR_2.11             ggplot2_3.5.1              
[25] edgeR_4.2.2                 limma_3.60.6                tictoc_1.2.1               

loaded via a namespace (and not attached):
  [1] tensorA_0.36.2.1        capushe_1.1.2           rstudioapi_0.17.1       jsonlite_1.8.9          magrittr_2.0.3          estimability_1.5.1     
  [7] farver_2.1.2            corrplot_0.95           ragg_1.3.3              fs_1.6.5                zlibbioc_1.50.0         vctrs_0.6.5            
 [13] memoise_2.0.1           askpass_1.2.1           rstatix_0.7.2           htmltools_0.5.8.1       S4Arrays_1.4.1          plotrix_3.8-4          
 [19] broom_1.0.7             SparseArray_1.4.8       Formula_1.2-5           gridGraphics_0.5-1      KernSmooth_2.23-26      htmlwidgets_1.6.4      
 [25] emmeans_1.11.0          cachem_1.1.0            mime_0.12               lifecycle_1.0.4         pkgconfig_2.0.3         Matrix_1.7-3           
 [31] R6_2.5.1                fastmap_1.2.0           GenomeInfoDbData_1.2.12 shiny_1.10.0            digest_0.6.37           HTSFilter_1.44.0       
 [37] colorspace_2.1-1        DESeq2_1.44.0           pkgload_1.4.0           HTSCluster_2.0.11       textshaping_1.0.0       ellipse_0.5.0          
 [43] labeling_0.4.3          httr_1.4.7              abind_1.4-8             compiler_4.5.0          remotes_2.5.0           proxy_0.4-27           
 [49] withr_3.0.2             backports_1.5.0         BiocParallel_1.38.0     carData_3.0-5           pkgbuild_1.4.6          ggsignif_0.6.4         
 [55] MASS_7.3-65             bayesm_3.1-6            sessioninfo_1.2.3       DelayedArray_0.30.1     scatterplot3d_0.3-44    gtools_3.9.5           
 [61] caTools_1.18.3          flashClust_1.01-2       tools_4.5.0             httpuv_1.6.15           glue_1.8.0              promises_1.3.2         
 [67] grid_4.5.0              cluster_2.1.8.1         generics_0.1.3          gtable_0.3.6            class_7.3-23            tidyr_1.3.1            
 [73] car_3.1-3               XVector_0.44.0          ggrepel_0.9.6           pillar_1.10.1           stringr_1.5.1           yulab.utils_0.2.0      
 [79] later_1.4.1             splines_4.5.0           robustbase_0.99-4-1     lattice_0.22-7          compositions_2.0-8      tidyselect_1.2.1       
 [85] locfit_1.5-9.12         miniUI_0.1.1.1          gridExtra_2.3           statmod_1.5.0           DEoptimR_1.1-3-1        DT_0.33                
 [91] stringi_1.8.4           UCSC.utils_1.0.0        qpdf_1.3.5              codetools_0.2-20        tibble_3.2.1            multcompView_0.1-10    
 [97] cli_3.6.3               systemfonts_1.2.1       xtable_1.8-4            munsell_0.5.1           Rcpp_1.0.14             leaps_3.2              
[103] ellipsis_0.3.2          Rmixmod_2.1.10          profvis_0.4.0           urlchecker_1.0.1        bitops_1.0-9            mvtnorm_1.3-3          
[109] scales_1.3.0            e1071_1.7-16            purrr_1.0.4             crayon_1.5.3            rlang_1.1.5

[vjcitn]

Have you looked at ?multicoreWorkers

 workers: 'integer(1)' Number of workers. Defaults to the maximum of 1
          or the number of cores determined by 'detectCores' minus 2
          unless environment variables
          'R_PARALLELLY_AVAILABLECORES_FALLBACK' or
          'BIOCPARALLEL_WORKER_NUMBER' are set otherwise.

With 48 cores, it is possible that 46 workers are activated by default.
Even if your base process is modest in size you will replicate it by
the number of workers. and 96Gb will not be enough. You should
see if cutting the worker number to 10 or so is is tolerable from an
overall RAM perspective. If the throughput is not acceptable then
other strategies can be considered. Let us know. (If you already tried
cutting down from the default number of workers assigned, then this
suggestion is nugatory.)