PiCRUSt2 - metagenome_pipeline.py : process killed after 3-4 hours.

[Mario Laterrière]

Hello Mr. Douglas,

there might be a memory leak in PiCRUSt2. At the metagenome prediction step, memory and swap usage is slowly rising up, and after a few hours, they both reach 100% and then the process crash displaying this error:

metagenome_pipeline.py: 45594 Killed  metagenome_pipeline.py -i ../out/6_otu_table.tsv -m ../out/16S_predicted.tsv -f ../out/EC_predicted.tsv -p 4 -o ../out/metagenome_out

My files are here: https://drive.google.com - PiCRUSt2 Files

Thank you.

Mario

[Gavin Douglas]

Hi Mario,

Sorry you’re running into this issue, but thanks for sending me your input files. I’ll have a chance to troubleshoot this on Monday. How much RAM do you have?

Best,

Gavin

--
You received this message because you are subscribed to the Google Groups "picrust-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to picrust-user...@googlegroups.com.
To post to this group, send email to picrus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mario Laterrière]

64 Go

Regards,

Mario

[Mario Laterrière]

Hello Mr. Douglas,

This is just to let you know that the problem is still there with the new version (2.0.0 b4).

[Gavin Douglas]

Hi Mario,

Yes I’m currently working on this problem, but the recent changes I pushed were not related.

The issue is that the number of sequence (i.e. organism) and gene family combinations explodes when there is a large number of input sequences (~40,000 in your case). The stratified output table takes up a lot of RAM (the final output file is 17 GB for your dataset) so one thing I’m working on is making this output file optional.

Best,

Gavin

[Gavin Douglas]

Hi again Mario,

You should be able to get the unstratified output with PICRUSt2 now very quickly with the latest version. If you use the new “—strat_out" option you can also try getting the stratified output, but you will likely run into the same memory issues as before. See the updates here: https://github.com/picrust/picrust2/pull/30

Best,

Gavin