memory error even with low rpb

[Cory Bishop]

Hello,

I've been trying to run the following command, but keep running out of memory.

I'm aware that this step is memory intensive, but I even reduced rpb to the absurdly low value of 1 and still got an error.  Same, whether I use --p-n-jobs 1 or -1

I'm running an i7 processor w/ 16GB RAM and have run numerous feature-classifier commands with Illumina MiSeq data using  --p-n-jobs 1 with no issue, so this seems like a different kind of problem.

qiime feature-classifier classify-sklearn \
   --i-reads dada2_output/representative_sequences.qza \
   --i-classifier $PWD/silva-138-99-nb-classifier.qza \
   --p-n-jobs -1 \
   --p-reads-per-batch 10 \
   --output-dir taxa

thoughts?

Cory

[Andre Comeau]

Might be a long shot, but I know (via PM) that you did a new install of your VBox image and so did you forget to change the "memory allowed" for the VBox program/image? Sometimes the default that it allows it access to is very small instead of given it access to your whole RAM and so could be causing the out of memory error.

	ANDRÉ M. COMEAU, PhD Manager • Integrated Microbiome Resource (IMR) T: 902.494.2684 | E: andre....@dal.ca	​Address for deliveries: Dept. of Pharmacology Tupper Med. Bldg., room 5D Dalhousie University 5850 College St. Halifax NS B3H 4R2
​	Research Associate (Lab Manager) Morgan Langille Lab • Dept. of Pharmacology ResearchGate Profile • GoogleScholar Publications
"Without fantasy, there is no science. Without fact, there is no art." - Nabokov "The good thing about science is that it's true whether or not you believe in it." - Neil deGrasse Tyson

---

From: microbio...@googlegroups.com <microbio...@googlegroups.com> on behalf of Cory Bishop <symbio...@gmail.com>
Sent: Thursday, August 26, 2021 11:29 AM
To: Microbiome Helper <microbio...@googlegroups.com>
Subject: [microbiome-helper] memory error even with low rpb

 

CAUTION: The Sender of this email is not from within Dalhousie.

--
You received this message because you are subscribed to the Google Groups "Microbiome Helper" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microbiome-hel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microbiome-helper/405a09ce-7573-4f60-8f36-1564c77330b4n%40googlegroups.com.

[Cory Bishop]

Not a bad guess, but no, I had set memory allocation as high as I could with my system (12GB, 2 cores). The only other change to the command is that I changed the path to the classifier, but can't see how that would affect anything. Any other troubleshooting advice?
C

[Cory Bishop]

By way of follow up:

I ran the command on my research cpu (6 core; 128GB RAM) with no limitation on reads per batch and using 2 cores and it ran fine. However, that cpu had been upgraded to qiime2-2021.4, natively installed on Linux OS. So hard to say whether due to version or power or type of install, but I still find it odd that even with rpb = 1 my laptop choked....I'll putter away and update this thread if I find something interesting out.

thanks for your help.

CB

On Thursday, August 26, 2021 at 12:21:40 PM UTC-3 André Comeau wrote:

[Andre Comeau]

Hmm...16 Gb might be OK for the SILVA ref dbase (note some of the QIIME2 forum answers are saying it is not, but you seem to have done so before)...and I assume your rep seq QZA file wasn't that enormous? One note of caution below in the code block you mention using: did you actually put --p-n-jobs -1 instead of 1? "Minus one" means "use all CPUs you have", in which case it would parallelize your job into 2-4 parallel batches (depending on how you CPU is multithreaded) and then try to load the SILVA dbase more than one time all at once which would kill your memory. Make sure it was a "1" only and see if that still causes the error. If so, then I'm not sure where to go next as it is tough to diagnose/bypass memory errors.

	ANDRÉ M. COMEAU, PhD Manager • Integrated Microbiome Resource (IMR) T: 902.494.2684 | E: andre....@dal.ca	​Address for deliveries: Dept. of Pharmacology Tupper Med. Bldg., room 5D Dalhousie University 5850 College St. Halifax NS B3H 4R2
​	Research Associate (Lab Manager) Morgan Langille Lab • Dept. of Pharmacology ResearchGate Profile • GoogleScholar Publications
"Without fantasy, there is no science. Without fact, there is no art." - Nabokov "The good thing about science is that it's true whether or not you believe in it." - Neil deGrasse Tyson

From: microbio...@googlegroups.com <microbio...@googlegroups.com> on behalf of Cory Bishop <symbio...@gmail.com>

Sent: Thursday, August 26, 2021 2:48 PM
To: Microbiome Helper <microbio...@googlegroups.com>
Subject: Re: [microbiome-helper] memory error even with low rpb

 

To view this discussion on the web visit https://groups.google.com/d/msgid/microbiome-helper/017ffb96-533a-450b-a3f6-272dcf1de3b1n%40googlegroups.com.