Pick Open Reference OTU's with SILVA is not generating data...?

Skip to first unread message

Crystal Weaver

Oct 5, 2016, 7:29:38 PM10/5/16
to Qiime 1 Forum
Hi all,

I'm attempting to pick open reference OTU's with SILVA for my 16s data. I used the following code:

qiime@qiime-190-virtual-box:~/Desktop$ pick_open_reference_otus.py -i ALL/seqsfna/seqs.fna -p param.txt -r SILVA/SILVA123_QIIME_release/rep_set/rep_set_all/97/97_otus.fasta -o ALL/SILVA_open_ref_otus -f -aO 4

The process hasn't bounced back an error yet, but it doesn't seem to be doing anything either. (My CPU and RAM usage is abnormally low, and the many output files are still at 0kb.)

My log file is below:

Logging started at 20:38:13 on 03 Oct 2016
QIIME version: 1.9.1

qiime_config values:
blastmat_dir    /qiime_software/blast-2.2.22-release/data
pick_otus_reference_seqs_fp    /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
sc_queue    all.q
pynast_template_alignment_fp    /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.pynast.fasta
cluster_jobs_fp    start_parallel_jobs.py
assign_taxonomy_reference_seqs_fp    /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
torque_queue    friendlyq
jobs_to_start    1
denoiser_min_per_core    50
assign_taxonomy_id_to_taxonomy_fp    /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt
temp_dir    /tmp/
blastall_fp    /qiime_software/blast-2.2.22-release/bin/blastall
seconds_to_sleep    1

parameter file values:
align_seqs.py:template_fp    SILVA/SILVA123_QIIME_release/core_alignment/core_alignment_SILVA123.fasta
filter_alignment:allowed_gap_frac    0.80
filter_alignment:entropy_threshold    0.10
filter_alignment:suppress_lane_mask_filter    True
assign_taxonomy:reference_seqs_fp    SILVA/SILVA123_QIIME_release/rep_set/rep_set_all/99/99_otus.fasta
assign_taxonomy:id_to_taxonomy_fp    SILVA/SILVA123_QIIME_release/taxonomy/taxonomy_all/99/taxonomy_7_levels.txt
parallel:jobs_to_start    4
pick_otus:enable_rev_strand_match    True

Input file md5 sums:
ALL/seqsfna/seqs.fna: 4dc1fc6f021648643813d38c5924bb36
SILVA/SILVA123_QIIME_release/rep_set/rep_set_all/97/97_otus.fasta: c62b7ca7818d058f40b2977af53a81b1

Executing commands.

# Pick Reference OTUs command
parallel_pick_otus_uclust_ref.py -i ALL/seqsfna/seqs.fna -o ALL/SILVA_open_ref_otus/step1_otus -r SILVA/SILVA123_QIIME_release/rep_set/rep_set_all/97/97_otus.fasta -T --jobs_to_start 4 --enable_rev_strand_match

(Yes, it's been running for nearly two days, but pick open reference OTU's normally takes 3 days on my machine when I go with the default method.)

Any suggestions for being able to check on progress? Or speeding up the process? (I have a whopping 10GB of RAM on this machine. Wish me luck...!)


Colin Brislawn

Oct 6, 2016, 12:42:44 PM10/6/16
to Qiime 1 Forum
Hello Crystal,

Thanks for getting in touch with us. Just to check, this data set has worked fine with qiime defaults, but now 'freezes / hangs' when silva is used, right? This difference greatly help us focus in on the problem, because we can infer that this difference is due to the change in database. 

Silva 99_otus.fasta database is 764MB, while greengenes 97 is 137 MB. This 5x increase of database will also greatly increase RAM usage. While 10 GB is enough for greengenes, it may not be enough for SILVA 99. You could try using SILVA 97, which is smaller, or silva rep_set_16S_only instead of rep_set_all which will only include 16S references. 

To confirm that RAM usage is causing this problem, you can use the linux command 'top' to view your available RAM and make sure you are not 'running out'. If you use all available RAM, this script will overflow into swap, slowing down the search to a crawl and creating the kind of freeze that you described.

Let me know if changing the database helps!

Crystal Weaver

Nov 1, 2016, 6:45:05 PM11/1/16
to Qiime 1 Forum
Hi Colin,

I'm so sorry it's taken me so long to respond. I was out of the country, and then it completely slipped my mind to tell you the great news... it worked! The RAM usage was the problem after all, and switching databases fixed all issues.

Can't thank you enough!!


Colin Brislawn

Nov 1, 2016, 7:07:34 PM11/1/16
to Qiime 1 Forum
Great! It's so satisfying when things work well.

If you have more questions, feel free to open another thread.


Reply all
Reply to author
0 new messages