kernel dynamic memory not freed after STAR run

[Ivan Molineris]

Hi everyone,
I have a strange problem on memory usage. I already asked about the problem on https://serverfault.com/questions/896282/cannot-explain-memory-occupancy-in-linux/896525#896525
and on https://stackoverflow.com/questions/48784781/what-is-kernel-dynamic-memory-in-linux-systems-and-how-to-free-it

I think that this problem is related to STAR since it happen after STAR runs but I'm not pretty sure that something in the STAR code is actually involved.
After a run of STAR not all the RAM used is released, even if the STAR processes end.

$ free -g
              total        used        free      shared  buff/cache   available
Mem:            125          67          56           0           1          57
Swap:           119           0         119

So I have only 57G available, but ps show that there is no process consuming memory

$ ps aux --sort -rss | head
USER        PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        987  0.0  0.0  54284 15932 ?        Ss   Feb09   0:03 /lib/systemd/systemd-journald
imoline+  49734  0.0  0.0  24640  8672 pts/0    Ss+  Feb13   0:05 /bin/bash
rstudio+   4803  0.0  0.0 128608  7400 ?        Ssl  Feb09   0:39 /usr/lib/rstudio-server/bin/rserver
imoline+  53699  0.0  0.0  23352  7220 pts/2    Ss   Feb13   0:03 /bin/bash
imoline+  68563  0.0  0.0  23200  6876 pts/1    Ss   10:37   0:00 -bash
root          1  0.0  0.0 204660  6804 ?        Ss   Feb09   0:05 /sbin/init
root      68556  0.0  0.0  99308  6800 ?        Ss   10:37   0:00 sshd: imolineris [priv]
root       4770  0.0  0.0  69944  6484 ?        Ss   Feb09   0:00 /usr/sbin/sshd -D

STAR is not running

$ ps aux --sort -rss | grep -i star

I try to unload the genome in shared memory, even if in the previous runs I do not used --genomeLoad

$ STAR --genomeLoad Remove --genomeDir /data/bioinfotree/task/annotations/dataset/gencode/hsapiens/27/GRCh38.primary_assembly.star_index/2.5.4a/
Feb 14 11:03:17 ..... started STAR run
Feb 14 11:03:17 ..... loading genome

EXITING: Did not find the genome in memory, did not remove any genomes from shared memory

Feb 14 11:03:17 ...... FATAL ERROR, exiting

Smem tell me that there is a lot of kernel dynamic memory in use, but I do not know what it means.

$ sudo smem -t -w -p

[sudo] password for imolineris:

Area                           Used      Cache   Noncache
firmware/hardware             0.00%      0.00%      0.00%
kernel image                  0.00%      0.00%      0.00%
kernel dynamic memory        54.68%      1.25%     53.43%
userspace memory              0.06%      0.03%      0.03%
free memory                  45.26%     45.26%      0.00%
----------------------------------------------------------

                            100.00%     46.54%     53.46%

Do you think that this problem is related to STAR?
Do you know how to free this memory without rebooting?

Thanks.

[Alexander Dobin]

Hi Ivan,

this is not supposed to happen, if a process does not exist, it cannot keep using non-shared memory, the OS should take care of reclaiming all of the RAM pages used.

Could you run free before and after STAR run? This will tell us if this is STAR's fault. :)

As far as I remember, kernel memory is allocated by OS for some kind of caching, and should be freed as your applications start using more RAM. Have you tried running multiple versions of STAR to fill up your RAM? Say 3 human genome should take ~90GB.

Cheers

Alex

[Ivan Molineris]

Hi Alex,
first of all thank you very much for your great works and for the support.
After many run of STAR on the same machine I'm pretty sure that the problem is something related to STAR, that consume the memory by allocating kernel dynamic memory that is not released after the STAR process finish.

The following is the output of sudo smem -t -w -p after few running of STAR_2.5.3a_modified, starting from a situation in which the ram is completely free, and after all STAR processes are completed correctly

Area                           Used      Cache   Noncache
firmware/hardware             0.00%      0.00%      0.00%
kernel image                  0.00%      0.00%      0.00%

kernel dynamic memory        46.28%      0.16%     46.12%
userspace memory              2.04%      0.03%      2.01%
free memory                  51.67%     51.67%      0.00%
----------------------------------------------------------
                            100.00%     51.87%     48.13%

The total ram is 128GB.

Note that the kernel dynamic memory is not seen as a cache by free or by htop.

Please find attached a Log.out file, all my processes are quite similar.

In order to free the ram I need to do
# echo 2 > /proc/sys/vm/drop_caches
or to reboot.

Can be this problem somehow related to the chancing of the genome? I do not specify any --genomeLoad option.

I will try to see if running subsequent STAR processes I end up swapping.

Cheers
Ivan

[Alexander Dobin]

Hi Ivan,

STAR uses standard gcc system calls to allocate memory for the genome. When the STAR job exits or even crashes, the OS should be able to recover the memory.

Do you see performance drop for any other application you start after STAR?

As far as I understand drop_caches, it does exactly that - drop the cache pages that are not currently in use. However, OS will also drop the cache as soon as it needs memory for a new job.

Cheers

Alex

[Ivan Molineris]

Hi Alex,
I mark this thread as complete because most probably the problem is someway related to the Linux kernel, the zfs module, the hardware, or some combination of these.
The solution I found is to periodically force the drop_chaces
Thanks