R eats up all RAM, then all swap, and the suddenly closes

Question

I am using the mediation package to run a mediation analysis.

But when I call the mediate function, R eats up all RAM, then it eats up all swap, and then RStudio suddenly closes.

The same happens if I call Rscript from the terminal, and the terminal suddenly closes.

Here is measures for RAM and swap. Notice the peaks:

This happens both with the R 4.0.5 version that comes with the Fedora repository (which uses Open BLAS), both with a custom R 4.1.0 version I compiled from source against Intel MKL.

What is the cause of this, and how I can debug?

Here is my current sessionInfo():

> sessionInfo()
R version 4.1.0 (2021-05-18)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Fedora 34 (Workstation Edition)

Matrix products: default
BLAS/LAPACK: /opt/intel/oneapi/mkl/2021.2.0/lib/intel64/libmkl_gf_lp64.so.1

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_GB.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_GB.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_GB.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.1.0 tools_4.1.0  

Answer 1

If the application needs all RAM and all virtual RAM you definitely hit the hard limit of your computer. If the application now requests more RAM from the OS, that request has to fail. How do you expect the application to deal with it?

As we do not know why more RAM was requested (it could be a memory leak, it could be that the algorithm just requires that much) we cannot distinguish bug or feature, but we can try things around that.

One quite cheap thing to try is to increase virtual memory size. Execution will get slower (disk access is measured in ms as opposed to RAM access times measured in ns), but maybe you find a value that the application can complete the task with.

If you cannot find any limit on virtual memory and the application always crashes due to the out-of-memory situation, you are likely facing a bug.

In any case you may want to talk to the vendor about this perception.