Plink2 out of memory for large VCFs

[kaustubh...@gmail.com]

Hi Chris,

I'm working with some large VCF.gz files of sequence data from SGDP. The per-sample files are around 25 GB.  These are 'full' vcfs containing both variant and non-variant calls, hence so big.  I tried to run Plink2 --vcf with 300 GB ram on the cluster, but plink ran out of memory after scanning 2147483k variants. I copy the relevant parts of the log file below. The whole log file is 70MB large after scanning  2147483k variants, so I'm not attaching the whole thing.

I'm wondering if plink could avoid running out of memory by writing temporary files to disk, as it seems to do for some other operations?

Thanks,

Kaustubh.

PLINK v2.0.0-a.7.5LM 64-bit Intel (10 Sep 2026)
Options in effect:
  --make-pgen
  --out LP6007075-DNA_A01
  --threads 1
  --vcf ../LP6007075-DNA_A01.annotated.nh2.vcf.gz

Random number seed: 1789275929
3031132 MiB RAM detected, ~2968861 available; reserving 1515566 MiB for main
workspace.
Allocated 269736 MiB successfully, after larger attempt(s) failed.
Using 1 compute thread.

Error: Out of memory.  The --memory flag may be helpful.

[DAVID J Cutler]

Kaustubh,

2^31 = 2,147,483,648 = 2,147,483k is the largest signed 32-bit integer.

I am about to make a conjecture that the maximum number of variants plink2
can handle is 2,147,483,648, and that it would take a very significant
rewrite to
change that limit.

I think it would be easier for you to prune all the non-variant sites.

Cheers,
dave

> --
> You received this message because you are subscribed to the Google Groups "plink2-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to plink2-users...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/plink2-users/4b8732e3-73b6-42cf-8c69-cb400707c8ban%40googlegroups.com.

[kaustubh...@gmail.com]

Interesting, thanks Dave, I hadn't thought about that! Could well be. I will probably extract only a subset of sites. I don't want to prune the non-variant sites because these are single-sample vcfs, by pruning non-variants I'd lose useful genotype information about a lot of loci.

Cheers,

Kaustubh.

[Christopher Chang]

Oh, it's suppose to print "Error: plink2 does not support more than 2^31 - 3 variants.  We recommend using other software for very deep studies of small numbers of genomes." in this situation, but there was an off-by-one error in the logic for printing that error message.  I will fix this in tomorrow's builds.