Excessive memory usage, the system automatically kills the process

[Chen Wang]

Dear author, thank you for your previous response to me. In your reply, you mentioned that reducing the number of groups in the popmap can decrease memory usage. But in my popmap, there are only two groups: pop1 (262 individuals) and pop2 (1325 individuals). Even if I execute pop1 and pop2 separately (popmap1 only has 262 individuals of pop1 and only 1 pop, and popmap2 only has 1325 individuals of pop2 and only 1 pop), the calculation of popmap1 can yield results, but in programs running with popmap2, the program will still be automatically killed by the system due to excessive memory usage. Is it because the number of individuals in 1325 is too large? Are there any other solutions available? Thank you for your help.

[Catchen, Julian]

Hi, you will likely need a computer with more memory, since 62Gb is quite limited for 1587 samples. If this is a denovo analysis, you could also try reducing the --batch-size parameter, which will cause the program to load fewer loci into memory at a time (default is 10k loci), however, the software still needs to calculate statistics across all 1587 individuals. Your previous message indicated that you were also trying to do smoothing, which implies you have aligned data, in which case you have to load in one scaffold or chromosome of data at a time (and 1587 individuals across that chromosome) to enable smoothing.  

 

From: 'Chen Wang' via Stacks <stacks...@googlegroups.com>
Date: Wednesday, August 21, 2024 at 2:34 PM
To: Stacks <stacks...@googlegroups.com>
Subject: [stacks] Excessive memory usage, the system automatically kills the process

Dear author, thank you for your previous response to me. In your reply, you mentioned that reducing the number of groups in the popmap can decrease memory usage. But in my popmap, there are only two groups: pop1 (262 individuals) and pop2 (1325 individuals). Even if I execute pop1 and pop2 separately (popmap1 only has 262 individuals of pop1 and only 1 pop, and popmap2 only has 1325 individuals of pop2 and only 1 pop), the calculation of popmap1 can yield results, but in programs running with popmap2, the program will still be automatically killed by the system due to excessive memory usage. Is it because the number of individuals in 1325 is too large? Are there any other solutions available? Thank you for your help.