MacBook Pro runs out of application memory when I run pick_closed_reference_otus against Greengenes

[Dani Haze]

I have the latest MacBook Pro with the following specs:

Processor: 2.9Ghz Intel Core i7
Memory: 16GB 2133 MHz LPDDR3

When I run the following command inside MacQIIME:

pick_closed_reference_otus.py -i combined_seqs.fna -o RESULTS -r gg_13_5.fasta -p params.txt -t gg_13_5_taxonomy.txt --parallel --jobs_to_start=2

... my Mac runs out of application memory and freezes.

The params.txt file contains the following:

"pick_otus:enable_rev_strand_match True
pick_otus:similarity 0.97
"
My combined_seqs.fna file consists of 6,153,481 sequences, but I have no trouble when running the exact same against Silva (https://www.arb-silva.de/).
The Greengenes files (gg_13_5.fasta and gg_13_5_taxonomy.txt) were downloaded from http://greengenes.secondgenome.com/downloads/database/13_5

How can I prevent this from happening? Should I split the combined_seqs.fna file somehow? Should I change any settings in my Mac? Am I using the correct Greengenes files?
I'd really appreciate if someone could shed some light! Many thanks!

Daniel

[Ana]

Try changing the amount of memory allocated to this script. I run QIIME on a server and use the "-M 1600" flag when I pick otus. You can adjust the memory allocation depending on your system. I hope this helps you!

[Colin Brislawn]

Hello Daniel,

Thanks for telling us about how you are running your script! You mentioned that:

The params.txt file contains the following:
"pick_otus:enable_rev_strand_match True

That will double your RAM usage. Try removing that line, or setting it to False.

Colin

PS How large is your seqs.fna file? If it's 100 GB, this is going to be a challenge to run on your laptop. If it's 10 GB, we can come up with something!

[Dani Haze]

Hi guys, many thanks! It is only 2.5GB, let me try the "-M 1600" option first and see if just that does the trick, otherwise I set the reverse strand match to False and see. Let me get back to you when I try the options! Thanks!!

[Dani Haze]

Do you mean "-M 1600" in the pick_closed_reference_otus.py call? I get "Error, no such option: -M"

[Ana]

Ah, I see now that -M is not an option with that particular script. I usually use that with pick_otus.py (where the -M is an option). Did you set reverse strand match to false and were you able to run the command successfully?

[Dani Haze]

Yes, I set the reverse strand to False, and it seems that the script has completed successfully, I run some script to check the mapping rate and it is high for all samples, so there is no need to run with a True argument. That should do it for the moment. Thanks guys!

[Colin Brislawn]

Great teamwork folks!

Feel free to open a new thread if you have more questions. This community is always here to help.

Colin