populations module is killed prematurely

[Austin Koontz]

Hi all, 

I'm using the Stacks (v2.59) populations module to process a reference-aligned dataset generated using gstacks. But every time I fire off the populations run, the run waits for a long while, and then is killed without error. I've attached the printed output (nohup.out), the populations.log file, and (just for good measure) the gstacks.log file from the prior gstacks run. 

I've been able to run the populations module on the same dataset when specifying the -R 80 missing data filter, so I suspect this is some kind of memory issue... the populations run that is failing specifies no missing data filter; I realize this will generate a ton of loci, but the impact of missing data is one of the factors I'm trying to examine, hence the absence of an R value.

I'm running using 30 cores on an Ubuntu (v18.04.6 LTS) server. If anyone has any suggestions on what could be causing this behavior, I'd really appreciate it. Thanks!

-Austin

[Catchen, Julian]

Hi Austin,

 

How much memory does your system have, or, how much are you requesting from the scheduler? The populations program will attempt to load one chromosome worth of loci into memory at a time. Gstacks says it built ~3 million loci, so that is quite a lot it needs to load into memory. Your mean coverage is quite low (~5x) with a lot of PCR duplicates having been removed (91%), so this will reduce the power of the analysis. If you can’t add more memory, a modest filter is likely to remove a lot of the loci (e.g. --min-mac 3 or -r 0.5) which may enable the procedure to complete if you can’t get more memory.

 

Best,

 

julian

[Austin Koontz]

Hi Julian, 

Thanks for you response. The system has 125 GB in memory (and 15 GB in "swap" memory); I'm running the program directly (no scheduler). I don't know if there's anything I can do to increase the amount of memory available, so I'll consider using more filters. I was avoiding filters in order to make this dataset comparable to a different species we're analyzing without any filters (but which uses a more distantly related reference genome--hence fewer loci!).

Thanks again,

-Austin

[Catchen, Julian]

That should be more than enough memory. I would monitor the job with top or run it within the time program:

 

/usr/bin/time -v populations …..

 

Which will tell you its max memory usage prior to exiting.

--
Stacks website: http://catchenlab.life.illinois.edu/stacks/
---
You received this message because you are subscribed to the Google Groups "Stacks" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stacks-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/stacks-users/c329f8dd-adb8-476d-89fa-00123297f823n%40googlegroups.com.

[Austin Koontz]

Thanks for this tip, Julian. I realize this is not really in the purview of the Stacks software, but I attached the output here for reference to others. From what I can tell, the memory usage when this crash is happening (Maximum resident set size?) is 130,403,496 KB, or ~130 GB, which does exceed the system's memory. 

I could try and address this by reducing the outputs being asked of the populations module (i.e. Fst, phylip, etc.), but I'm not sure that would help. 

Thanks again for your suggestion,

-Austin

[Catchen, Julian]

Hi Austin,

 

Very interesting, I would not have expected that. You could try using the --batch-size [int] option to populations, which will limit the number of loci it tries to run per batch. By default, it will load one chromosome at a time into memory, but you can limit to 5 or 10k loci (or whatever) at a time, which will hopefully address the memory issue. This option is not super well tested for reference data (it is used regularly in de novo data) but it should work.

[Austin Koontz]

Thanks for the suggestion. I tried specifying --batch-size values of 10,000 and 5,000, but both resulted in the same error (see attached). The memory usage was roughly the same when the process was killed, so I wonder if this argument isn't being registered somehow?