Error during "assign_taxonomy.py -m rdp -c 0.8" with Silva123, 18S

[Hiunvun Chin]

Hi,
Here are the scripts and errors, it is grateful to have your suggestion how I can solve this problem.

qiime@qiime-190-virtual-box:~$ assign_taxonomy.py -i rep_set.fna -r 97_otus_18S.fasta -t taxonomy_all_levels.txt -o taxonomy_results_rdp_confid_0.8/ -m rdp -c 0.8

Traceback (most recent call last):
  File "/usr/local/bin/assign_taxonomy.py", line 417, in <module>
    main()
  File "/usr/local/bin/assign_taxonomy.py", line 394, in main
    log_path=log_path)
  File "/usr/local/lib/python2.7/dist-packages/qiime/assign_taxonomy.py", line 860, in __call__
    max_memory=max_memory, tmp_dir=tmp_dir)
  File "/usr/local/lib/python2.7/dist-packages/bfillings/rdp_classifier.py", line 515, in train_rdp_classifier_and_assign_taxonomy
    tmp_dir=tmp_dir)
  File "/usr/local/lib/python2.7/dist-packages/bfillings/rdp_classifier.py", line 485, in train_rdp_classifier
    return app(training_seqs_file)
  File "/usr/local/lib/python2.7/dist-packages/bfillings/rdp_classifier.py", line 327, in __call__
    remove_tmp=remove_tmp)
  File "/usr/local/lib/python2.7/dist-packages/burrito/util.py", line 285, in __call__
    'StdErr:\n%s\n' % open(errfile).read())
burrito.util.ApplicationError: Unacceptable application exit status: 137
Command:
cd "/home/qiime/"; java -Xmx4000M -cp "/qiime_software/rdpclassifier-2.2-release/rdp_classifier-2.2.jar" edu.msu.cme.rdp.classifier.train.ClassifierTraineeMaker "/tmp/RdpTaxonomy_6RLRvN.txt" "/tmp/tmppfPMc4K98y4ryALpvCi1.txt" 1 version1 cogent "/tmp/RdpTrainer_uZg01k" > "/tmp/tmpOzzXG2w3fjVjhFbrvvzT.txt" 2> "/tmp/tmpuFiusKtnvryReXbiMx0I.txt"
StdOut:

StdErr:
Copyright 2006 Michigan State University Board of Trustees.

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA

Authors's mailng address:
Center for Microbial Ecology
2225A Biomedical Physical Science
Michigan State University
East Lansing, Michigan USA 48824-4320
E-mail: James R. Cole at co...@msu.edu
    Qiong Wang at wang...@msu.edu
    James M. Tiedje at tie...@msu.edu


Killed

The print_qiime_config.py -t also presented for your ref:

System information
==================
         Platform:    linux2
   Python version:    2.7.3 (default, Dec 18 2014, 19:10:20)  [GCC 4.6.3]
Python executable:    /usr/bin/python

QIIME default reference information
===================================
For details on what files are used as QIIME's default references, see here:
 https://github.com/biocore/qiime-default-reference/releases/tag/0.1.2

Dependency versions
===================
          QIIME library version:    1.9.1
           QIIME script version:    1.9.1
qiime-default-reference version:    0.1.2
                  NumPy version:    1.9.2
                  SciPy version:    0.15.1
                 pandas version:    0.16.1
             matplotlib version:    1.4.3
            biom-format version:    2.1.4
                   h5py version:    2.4.0 (HDF5 version: 1.8.4)
                   qcli version:    0.1.1
                   pyqi version:    0.3.2
             scikit-bio version:    0.2.3
                 PyNAST version:    1.2.2
                Emperor version:    0.9.51
                burrito version:    0.9.1
       burrito-fillings version:    0.1.1
              sortmerna version:    SortMeRNA version 2.0, 29/11/2014
              sumaclust version:    SUMACLUST Version 1.0.00
                  swarm version:    Swarm 1.2.19 [May 26 2015 13:50:14]
                          gdata:    Installed.

QIIME config values
===================
For definitions of these settings and to learn how to configure QIIME, see here:
 http://qiime.org/install/qiime_config.html
 http://qiime.org/tutorials/parallel_qiime.html

                     blastmat_dir:    /qiime_software/blast-2.2.22-release/data
      pick_otus_reference_seqs_fp:    /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
                         sc_queue:    all.q
      topiaryexplorer_project_dir:    None
     pynast_template_alignment_fp:    /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.pynast.fasta
                  cluster_jobs_fp:    start_parallel_jobs.py
pynast_template_alignment_blastdb:    None
assign_taxonomy_reference_seqs_fp:    /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
                     torque_queue:    friendlyq
                    jobs_to_start:    1
                       slurm_time:    None
            denoiser_min_per_core:    50
assign_taxonomy_id_to_taxonomy_fp:    /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt
                         temp_dir:    /tmp/
                     slurm_memory:    None
                      slurm_queue:    None
                      blastall_fp:    /qiime_software/blast-2.2.22-release/bin/blastall
                 seconds_to_sleep:    1

QIIME base install test results
===============================
.........
----------------------------------------------------------------------
Ran 9 tests in 0.027s

OK

I tried to add --rdp_max_memory 2000, 4000 or 12000 at the end of the script, but still not good.

Thank you very much!
Chin

[Stefan Janssen]

Hi.
exit status: 137 means that you ran out of memory and your process had to be killed therefore.

Try to manually increase the memory by changing the 4000M in the following to a higher number, given your computer has more than 4GB memory:

cd "/home/qiime/"; java -Xmx4000M -cp "/qiime_software/

rdpclassifier-2.2-release/rdp_classifier-2.2.jar" edu.msu.cme.rdp.classifier.train.ClassifierTraineeMaker "/tmp/RdpTaxonomy_6RLRvN.txt" "/tmp/tmppfPMc4K98y4ryALpvCi1.txt" 1 version1 cogent "/tmp/RdpTrainer_uZg01k" > "/tmp/tmpOzzXG2w3fjVjhFbrvvzT.txt" 2> "/tmp/tmpuFiusKtnvryReXbiMx0I.txt"

Do you see the 4000M value change if you provide --rdp_max_memory ?

A workaround might be to split your input file rep_set.fna into several chunks and process them independently. You would than need to combine all results into one after running RDP successfully.

[TonyWalters]

I'd also suggest looking into the notes text file that comes with the SILVA release and search for RDP, which has high memory requirements.

-Tony

[Hiunvun Chin]

Hi Stefan,
Thank you. I have a physical ram of 12 GB. Only set at about 2 GB in Virtual Box.
I can see the 4000M changed to 24000M, if i add --rdp_max_memory 24000, even i don't have a 24 GB physical ram.

Hi Tony, I checked the Silva 123 notes file, it seems rdp requires 24000 MB:

Taxonomic Assignment
16 gb required for UCLUST for assigning taxonomy with the 97% reference sequences/taxonomy mapping files. The 64 bit version of UCLUST is required for allocation of memory > 4 gigs.
For RDP, >24 gigs (--rdp_max_memory 24000) was required for taxonomic assignments with the 97% OTUs/taxonomy mapping files.

I guess i need to upgrade my ram from 12 GB to >24 GB (as less than 24 GB will be usable if install a 24 GB ram) in order to run the scripts?

Thank you very much.
Chin

[TonyWalters]

Hello,

You can try to allocate more memory (both for the VB and the command line) than physically present for RDP, but it may be problematic with the underlying java. See: https://www.quora.com/Can-Java-heap-have-a-size-larger-than-actual-physical-memory

As an alternative, you might try the uclust taxonomic assigner (also uses more memory than you have allocated for VB, but not as much as RDP).

Another alternative would be to use an Amazon instance to process this part of your data: http://qiime.org/1.9.0/tutorials/working_with_aws.html

[Hiunvun Chin]

Dear Tony,
Great. I will try to look into it.
Regards,
Chin

[Stefan Janssen]

you also might consider using our qiita platform: https://qiita.ucsd.edu/
The compute will be done on our servers, which have at least 256 GB RAM.

[Hiunvun Chin]

Hi Stefan,
Thank you for the information.
Chin