Out of Memory, Failed allocation size: ....

[davide m]

Hi all,

I am trying to run a GWAS but I am receiving an Out of memory error as below. Strangely, it looks like plink fails to allocate 361968896, which I suppose is ~350Mb, while having 4Gb available for main space.

For completeness, this is launched as a job on slurm through snakemake, assigning 8Gb of memory to the job, so there is another 4Gb available for "out of main space" operations. There are not overly long IDs, so unfortunately is not connected to that known issue. Does anyone came across this issue? Any suggestion to solve it?

Thank you very much in advance for your help.

See below the log:

PLINK v1.90b6.21 64-bit (19 Oct 2020)          www.cog-genomics.org/plink/1.9/
(C) 2005-2020 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to GWAS_results/UKBBtrain5000_BMI.chr21.scaffold_EUR.tmp.log.
Options in effect:
  --adjust
  --allow-no-sex
  --bfile GWASinput.chr21.filtered
  --ci 0.95
  --covar covar_files/covPLINK
  --fam GWASinput.chr21.filtered.phenos.fam
  --linear hide-covar
  --memory 4096
  --out GWAS_results/chr21

64222 MB RAM detected; reserving 4096 MB for main workspace.
8482 variants loaded from .bim file.
350745 people (0 males, 0 females, 350745 ambiguous) loaded from .fam.
Ambiguous sex IDs written to
GWAS_results/chr21.nosex .
350745 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
--covar: 127 covariates loaded.
Before main variant filters, 350745 founders and 0 nonfounders present.
Calculating allele frequencies... done.
8482 variants and 350745 people pass filters and QC.
Phenotype data is quantitative.

Error: Out of memory.  The --memory flag may be helpful.
Failed allocation size: 361968896

[Christopher Chang]

1. You are misunderstanding how --memory works.  >99% of the time, plink will only try to allocate from the --memory workspace.  It doesn't matter if you actually have >200000 MiB available; if you specify "--memory 4096", and 3900 of that 4096 MiB of workspace memory is already used, a 361968896 byte allocation attempt will fail at that point.

So if you really have 8192 MiB available, and you're starting a memory-constrained job, you should be specifying something like "--memory 8000".

2. As a practical matter, plink 2.0 is usually hundreds of times as fast as plink 1.9 at linear regression.