feature-classifier classify-sklearn job killed due to memory contraints

[mj]

Hi there, new QIIME user here following the Amplicon SOP.

Like many others, I'm running into issues when assigning taxonomy and I believe it's memory related. The job gets killed after 5 minutes and I've reduced --p-reads-per-batch --p-n-jobs. There is 16303708 kB of RAM on this linux box. I'm not sure what else I can do. Should I decrease --p-reads-per-batch 1000 even more? What would be a reasonable batch size?

qiime feature-classifier classify-sklearn
--i-classifier /home/Downloads/silva-138-99-nb-classifier.qza
--i-reads /home/Rocks/outputs/qza_intermediates/rocks16S_rep_seqs.qza
--o-classification rocks16S_taxonomy.qza
--p-reads-per-batch 1000
--p-n-jobs 1
--p-confidence 1
--verbose &> 16S_classify_verbose.log & disown

I downloaded the silva-138-99-nb-classifier.qza from the QIIME data resources page but I saw that on your Virtual Box Amplicon SOP (https://github.com/LangilleLab/microbiome_helper/wiki/Amplicon-SOP-v2-(qiime2-2022.11)) there is a trained classifier on the 16S V4/V5 region (classifier_silva_132_99_16S_V4.V5_515F_926R.qza), which is what we sequenced. Is it possible to download this externally without going through the virtual box?

Thank you.

[Andre Comeau]

Our trained classifiers are all here for individual download: http://kronos.pharmacology.dal.ca/public_files/MH/taxa_classifiers/

Note however, that the use of "region-specific" dbases is now not recommended due to a potential bug we found (discussed here: https://github.com/LangilleLab/microbiome_helper/issues/43).

Normally, with 16 Gb of RAM I would have expected you to be able to get through any reasonably-sized dataset, but memory issues are the most common problem at that step. What is the total number of ASVs in your rep_seqs file? Even if working with a large dataset of 1M reads, you should be still collapsing down to only 100-1000s of ASVs after Deblur (on the higher end if a very diverse environment), which should be classifiable.

	ANDRÉ M. COMEAU, PhD Manager • Integrated Microbiome Resource (IMR) T: 902.494.2684 | E: andre....@dal.ca	​Address for deliveries: Dept. of Pharmacology Tupper Med. Bldg., room 5D Dalhousie University 5850 College St. Halifax NS B3H 4R2
​	Research Associate (Lab Manager) Morgan Langille Lab • Dept. of Pharmacology ResearchGate Profile • GoogleScholar Publications
"Without fantasy, there is no science. Without fact, there is no art." - Nabokov "The good thing about science is that it's true whether or not you believe in it." - Neil deGrasse Tyson

---

From: microbio...@googlegroups.com <microbio...@googlegroups.com> on behalf of mj <meng...@gmail.com>
Sent: Thursday, March 30, 2023 3:45 PM
To: Microbiome Helper <microbio...@googlegroups.com>
Subject: [microbiome-helper] feature-classifier classify-sklearn job killed due to memory contraints

 

CAUTION: The Sender of this email is not from within Dalhousie.

--
You received this message because you are subscribed to the Google Groups "Microbiome Helper" group.
To unsubscribe from this group and stop receiving emails from it, send an email to microbiome-hel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/microbiome-helper/4bbb208f-6441-4390-964c-f1f3b82bf31en%40googlegroups.com.