error assign_taxonomy.py OSError: [Errno 39] tmp directory not empty

[Will Overholt]

Hello,

I've been working the past week on getting qiime installed on our cluster. Things were mostly going smoothly until I tried to assign taxonomy using RDP (same error for parallel_assign_taxonomy_rdp.py and assign_taxonomy.py).
Then I've been getting this error:

Traceback (most recent call last):
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/bin/assign_taxonomy.py"
, line 305, in <module>
    main()
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/bin/assign_taxonomy.py"
, line 283, in main
    result_path=temp_result_path,log_path=log_path)
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/assign_taxono
my.py", line 425, in __call__
    max_memory=max_memory, tmp_dir=tmp_dir)
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/pycogent_backports/rdp_classifier.py", line 517, in train_rdp_classifier_and_assign_taxonomy
    rmtree(training_dir)
  File "/nv/hp10/woverholt3/qiime1.6/python-2.7.3-release/lib/python2.7/shutil.py", line 254, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/nv/hp10/woverholt3/qiime1.6/python-2.7.3-release/lib/python2.7/shutil.py", line 252, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty: '/nv/hp10/woverholt3/data/qiime_files/tmp/RdpTrainer_L0QvBA'


I get the same error when I run the assign_taxonomy test script.

Here is the printout from print_qiime_config.py -t

System information
==================
         Platform:      linux2
   Python version:      2.7.3 (default, Dec 20 2012, 13:15:47)  [GCC 4.4.6 20110731 (Red Hat 4.4.6-3)]
Python executable:      /nv/hp10/woverholt3/qiime1.6/python-2.7.3-release/bin/python

Dependency versions
===================
                     PyCogent version:  1.5.3
                        NumPy version:  1.5.1
                   matplotlib version:  1.1.0
                  biom-format version:  1.1.1
                QIIME library version:  1.6.0
                 QIIME script version:  1.6.0
        PyNAST version (if installed):  1.2
RDP Classifier version (if installed):  rdp_classifier-2.2.jar
          Java version (if installed):  1.6.0_31

QIIME config values
===================
blastmat_dir:      /nv/hp10/woverholt3/qiime1.6/blast-2.2.22-release/data
sc_queue:      all.q
topiaryexplorer_project_dir:      None
pynast_template_alignment_fp:      /nv/hp10/woverholt3/qiime1.6/core_set_aligned.fasta.imputed
cluster_jobs_fp:      /nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/bin/start_parallel_jobs.py
pynast_template_alignment_blastdb:      None
assign_taxonomy_reference_seqs_fp:      /nv/hp10/woverholt3/qiime1.6/gg_otus-12_10-release/rep_set/97_otus.fasta
torque_queue:      friendlyq
qiime_test_data_dir:      None
template_alignment_lanemask_fp:      /nv/hp10/woverholt3/qiime1.6/lanemask_in_1s_and_0s
jobs_to_start:      1
cloud_environment:      False
qiime_scripts_dir:      /nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/bin
denoiser_min_per_core:      50
working_dir:      /nv/hp10/woverholt3/data/qiime_files/PB_beach/
python_exe_fp:      /nv/hp10/woverholt3/qiime1.6/python-2.7.3-release/bin/python
temp_dir:      /nv/hp10/woverholt3/data/qiime_files/tmp/
blastall_fp:      /nv/hp10/woverholt3/qiime1.6/blast-2.2.22-release/bin/blastall
seconds_to_sleep:      60
assign_taxonomy_id_to_taxonomy_fp:      /nv/hp10/woverholt3/qiime1.6/gg_otus-12_10-release/taxonomy/97_otu_taxonomy.txt

running checks:

test_FastTree_supported_version (__main__.Qiime_config)
FastTree is in path and version is supported ... ok
test_INFERNAL_supported_version (__main__.Qiime_config)
INFERNAL is in path and version is supported ... ok
test_ParsInsert_supported_version (__main__.Qiime_config)
ParsInsert is in path and version is supported ... ok
test_R_supported_version (__main__.Qiime_config)
R is in path and version is supported ... ok
test_ampliconnoise_install (__main__.Qiime_config)
AmpliconNoise install looks sane. ... ok
test_blast_supported_version (__main__.Qiime_config)
blast is in path and version is supported ... ok
test_blastall_fp (__main__.Qiime_config)
blastall_fp is set to a valid path ... ok
test_blastmat_dir (__main__.Qiime_config)
blastmat_dir is set to a valid path. ... ok
test_cdbtools_supported_version (__main__.Qiime_config)
cdbtools is in path and version is supported ... ok
test_cdhit_supported_version (__main__.Qiime_config)
cd-hit is in path and version is supported ... ok
test_chimeraSlayer_install (__main__.Qiime_config)
no obvious problems with ChimeraSlayer install ... ok
test_clearcut_supported_version (__main__.Qiime_config)
clearcut is in path and version is supported ... ok
test_cluster_jobs_fp (__main__.Qiime_config)
cluster_jobs_fp is set to a valid path and is executable ... ok
test_denoiser_supported_version (__main__.Qiime_config)
denoiser aligner is ready to use ... FAIL
test_for_obsolete_values (__main__.Qiime_config)
local qiime_config has no extra params ... ok
test_gdata_install (__main__.Qiime_config)
gdata is installed ... ok
test_matplotlib_suported_version (__main__.Qiime_config)
maptplotlib version is supported ... ok
test_mothur_supported_version (__main__.Qiime_config)
mothur is in path and version is supported ... ERROR
test_muscle_supported_version (__main__.Qiime_config)
muscle is in path and version is supported ... ok
test_numpy_suported_version (__main__.Qiime_config)
numpy version is supported ... ok
test_pplacer_supported_version (__main__.Qiime_config)
pplacer is in path and version is supported ... ok
test_pynast_suported_version (__main__.Qiime_config)
pynast version is supported ... ok
test_pynast_template_alignment_blastdb_fp (__main__.Qiime_config)
pynast_template_alignment_blastdb, if set, is set to a valid path ... ok
test_pynast_template_alignment_fp (__main__.Qiime_config)
pynast_template_alignment, if set, is set to a valid path ... ok
test_python_exe_fp (__main__.Qiime_config)
python_exe_fp is set to a working python env ... ok
test_python_supported_version (__main__.Qiime_config)
python is in path and version is supported ... ok
test_qiime_scripts_dir (__main__.Qiime_config)
qiime_scripts_dir, if set, is set to a valid path ... ok
test_qiime_test_data_dir (__main__.Qiime_config)
qiime_test_data_dir, if set, is set to a valid path ... ok
test_raxmlHPC_supported_version (__main__.Qiime_config)
raxmlHPC is in path and version is supported ... ok
test_rtax_supported_version (__main__.Qiime_config)
rtax is in path and version is supported ... ok
test_temp_dir (__main__.Qiime_config)
temp_dir, if set, is set to a valid path ... ok
test_template_alignment_lanemask_fp (__main__.Qiime_config)
template_alignment_lanemask, if set, is set to a valid path ... ok
test_uclust_supported_version (__main__.Qiime_config)
uclust is in path and version is supported ... ok
test_usearch_supported_version (__main__.Qiime_config)
usearch is in path and version is supported ... FAIL
test_working_dir (__main__.Qiime_config)
working_dir, if set, is set to a valid path ... ok

======================================================================
ERROR: test_mothur_supported_version (__main__.Qiime_config)
mothur is in path and version is supported
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/bin/print_qiime_config.py", line 514, in test_mothur_supported_version
    remove('mothur.log')
OSError: [Errno 2] No such file or directory: 'mothur.log'

======================================================================
FAIL: test_denoiser_supported_version (__main__.Qiime_config)
denoiser aligner is ready to use
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/bin/print_qiime_config.py", line 538, in test_denoiser_supported_version
    "which components of QIIME you plan to use.")
AssertionError: Denoiser flowgram aligner not found or not executable.This may or may not be a problem depending on which components of QIIME you plan to use.

======================================================================
FAIL: test_usearch_supported_version (__main__.Qiime_config)
usearch is in path and version is supported
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/bin/print_qiime_config.py", line 674, in test_usearch_supported_version
    "which components of QIIME you plan to use.")
AssertionError: usearch not found. This may or may not be a problem depending on which components of QIIME you plan to use.

----------------------------------------------------------------------
Ran 35 tests in 0.633s

FAILED (failures=2, errors=1)

Have you seen this before? I tried combing the forum for past issues but I didn't turn anything up.

Please let me know if you need any additional information! In the meantime I'll continue to use my laptop, but it would be really nice to have the full suite of scripts working on the biocluster (since multiple users are planning on using it).

Thanks for you time!

Best Regards,
Will

[Yoshiki Vázquez Baeza]

Hello Will,

We are working on this issue, we will get back to you ASAP.

Thanks!

Yoshiki.

2012/12/29 Will Overholt <waove...@gmail.com>

--
 
 
 

[Kyle Bittinger]

Will,

This is the author of the code appearing in your error message.  I have not pinpointed the problem yet, but I think you may be able to help me figure it out.

I'd like you to try the assign_taxonomy.py command again and send me one more piece of information.  At the bottom of your error message, there will be a line like:

OSError: [Errno 39] Directory not empty: '/nv/hp10/woverholt3/data/qiime_files/tmp/RdpTrainer_L0QvBA'

Would you run assign_taxonomy.py again, then send me a long listing of the files (ls -lR) in the directory triggering the error?  For example, in this case I'd like to see the output of:

ls -lR /nv/hp10/woverholt3/data/qiime_files/tmp/RdpTrainer_L0QvBA

Thank you!!

Kyle

--
 
 
 

[Will Overholt]

Hey Kyle,

Thank you for the incredibly prompt response, and I'm sorry for taking a while to respond. I wanted to make sure of the results before updating you.

It looks like the /tmp/RdpTrainer_XXXX file is empty. I've ran the test script and the regular script twice each and every time the folder it is complaining about is empty. The parallel test script is not working (keeps timing out) although I was pretty sure it worked yesterday when I was playing around with this problem, so I'm not sure what is going on there. The parallel script itself still gives the same Errno 39 and outputs the number of tmp/RdpTrainer_XXXX folders as number of jobs I started, but each is also empty.

I do get a folder called "jobs" in my working directory that is full of RDP_XXXX_## executable files.

For the assign_taxonomy.py script, my specified output directory is empty. For the parallel_assign_taxonomy_rdp.py script the output directory is filled with fasta files (0..num jobs started), deletion_list.txt, expected_out_files.txt, merge_map.txt, and RDP_YYYY_jobs.txt. 

As a side note, this is slightly different than I was getting yesterday. My tmp folder was initially the clusters /tmp/ folder, and I thought it might have been a read/write error. So I updated the qiime_config file with a local /tmp/ folder. The system tmp folders were populated with a few files (4 if I remember correctly). If it'll be helpful I try recreating those files (the tmp folder files with my username were cleared so I don't have access to them anymore).

Please et me know if there is anything else I can provide to help get to the bottom of this.

Best,

Will

--
 
 
 

--
Will Overholt
Georgia Institute of Technology, School of Biology
Room 218A, Cherry Emerson Building
310 Ferst Drive
Atlanta, GA 30332
(404) 385-4208
waove...@gatech.edu

[Kyle Bittinger]

Will,

Let's work on the non-parallel version first (assign_taxonomy.py).  If I can get that working for you, at least you have a way forward.

From what you say, it sounds like the system tmp folder is working better than your user temp folder.  Would you change the qiime_config back to its initial setting, re-run assign_taxonomy.py, and send me the output of

ls -laR tmp/RdpTrainer_XXXX

I have a couple more questions: what are the permissions of the temp folder specified in your qiime_config?  What are the user and group owners of this folder?  Also, when you submit you assign_taxonomy.py job to the cluster, under what username is the process executed?

Best,

Kyle

--
 
 
 

[Will Overholt]

Hey Kyle,

The system /tmp/ folder has full rwx access for owner/group/others (drwxrwxrwx). The files inside the folder have various permissions, everything with a root owner is only rw for root. Everything under my owner name (woverholt3), is under our labs group owner (bio-kostka) and I have rw access to them (-rw-r--r--).

When I submit the job to the cluster it is under my username. 

I changed the tmp default folder back to the systems /tmp/ folder, but the RdpTrainer_XXXX is empty.

I think the working directory was also originally set to /tmp/. So I changed that as well in qiime_config and re-ran the script and the new RdpTrainer_XXXX is still empty.

[woverholt3@iw-k40-22 PB_beach]$ assign_taxonomy.py -i test_rep_set100.fna -o test_assign_tax

Traceback (most recent call last):
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/bin/assign_taxonomy.py", line 305, in <module>
    main()
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/bin/assign_taxonomy.py", line 283, in main
    result_path=temp_result_path,log_path=log_path)

  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/assign_taxonomy.py", line 425, in __call__

    max_memory=max_memory, tmp_dir=tmp_dir)
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/pycogent_backports/rdp_classifier.py", line 517, in train_rdp_classifier_and_assign_taxonomy
    rmtree(training_dir)
  File "/nv/hp10/woverholt3/qiime1.6/python-2.7.3-release/lib/python2.7/shutil.py", line 254, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/nv/hp10/woverholt3/qiime1.6/python-2.7.3-release/lib/python2.7/shutil.py", line 252, in rmtree
    os.rmdir(path)

OSError: [Errno 39] Directory not empty: '/tmp/RdpTrainer_E06pw4'

[woverholt3@iw-k40-22 PB_beach]$ ls /tmp/RdpTrainer_E06pw4/

[woverholt3@iw-k40-22 PB_beach]$ ls -laR /tmp/RdpTrainer_E06pw4/
/tmp/RdpTrainer_E06pw4/:
total 40
drwx------ 2 woverholt3 bio-kostka  4096 Dec 30 10:22 .
drwxrwxrwt 7 root       root       36864 Dec 30 10:22 ..

I'm really at a loss as to what happened when I messed with the qiime_config file. I can try re-installing qiime1.6 to recreate the original qiime_config file if you think that will help?

Thanks!

Best,
Will

[Kyle Bittinger]

OK, I'm going to have you run some unit tests to gather some more information.  If this does not reveal the problem, I plan to change this error to a warning.  At least you can get some results, even if a temp directory is left for manual cleanup.  Issuing a warning may also help us gather more info.

For these tests, please set the temp_dir to "/nv/hp10/woverholt3/data/qiime_files/tmp/" in your .qiime_config.  I don't know much about the configuration of your cluster, but I'm guessing that any process you launch should have access to this directory.  Also, any leftover temp files won't be cleared away if the system is restarted.

From the main directory of the QIIME 1.6 release (the folder containing setup.py), please run the following commands and send me the output:

python tests/test_split_libraries.py

python tests/test_assign_taxonomy.py

python tests/test_parallel/test_assign_taxonomy.py

Thanks for your patience as we work this out,

Kyle

--
 
 
 

[Will Overholt]

Hey Kyle,

Thanks for all your help with this!

Here is the output from test_split_libraries.py
--------------------------------------------
Ran 35 tests in 16.679s

OK
--------------------------------------------

Output from test_assign_taxonomy.py

--------------------------------------------

ERROR: test_assignment_maximum_conf (__main__.MothurTaxonAssignerTests)

----------------------------------------------------------------------
Traceback (most recent call last):

  File "tests/test_assign_taxonomy.py", line 580, in test_assignment_maximum_conf
    result = assigner(self.seq_fp1)
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/assign_taxonomy.py", line 367, in __call__
    output_fp=result_path,
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/pycogent_backports/mothur.py", line 505, in mothur_classify_file
    result = app(query_file)
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/pycogent_backports/mothur.py", line 139, in __call__
    (exit_status, args))
ApplicationError: Unacceptable application exit status: 1, command: ['mothur', '#classify.seqs(fasta=/tmp/tmpKQxzymRmvUEoLCxICQSv.fasta, reference=/tmp/tmpgK9y15.ref.fa, taxonomy=/tmp/tmpq_HMMW.tax.txt, cutoff=95)']

======================================================================
ERROR: test_assignment_minimum_conf (__main__.MothurTaxonAssignerTests)

----------------------------------------------------------------------
Traceback (most recent call last):

  File "tests/test_assign_taxonomy.py", line 561, in test_assignment_minimum_conf
    result = assigner(self.seq_fp1)
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/assign_taxonomy.py", line 367, in __call__
    output_fp=result_path,
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/pycogent_backports/mothur.py", line 505, in mothur_classify_file
    result = app(query_file)
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/pycogent_backports/mothur.py", line 139, in __call__
    (exit_status, args))
ApplicationError: Unacceptable application exit status: 1, command: ['mothur', '#classify.seqs(fasta=/tmp/tmpQ97FGZnMNxu32HTOL0ss.fasta, reference=/tmp/tmp9G4A2b.ref.fa, taxonomy=/tmp/tmpmmuWFM.tax.txt, cutoff=0)']

======================================================================
ERROR: test_unassignable (__main__.MothurTaxonAssignerTests)

----------------------------------------------------------------------
Traceback (most recent call last):

  File "tests/test_assign_taxonomy.py", line 601, in test_unassignable
    result = assigner(self.seq_fp1)
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/assign_taxonomy.py", line 367, in __call__
    output_fp=result_path,
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/pycogent_backports/mothur.py", line 505, in mothur_classify_file
    result = app(query_file)
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/pycogent_backports/mothur.py", line 139, in __call__
    (exit_status, args))
ApplicationError: Unacceptable application exit status: 1, command: ['mothur', '#classify.seqs(fasta=/tmp/tmpidABiIDoQnZ0L6Bksa0G.fasta, reference=/tmp/tmpID2pky.ref.fa, taxonomy=/tmp/tmpPi_IbW.tax.txt, cutoff=80)']

======================================================================
ERROR: test_taxa_with_special_characters (__main__.RdpTaxonAssignerTests)
Special characters in taxa do not cause RDP errors

----------------------------------------------------------------------
Traceback (most recent call last):

  File "tests/test_assign_taxonomy.py", line 695, in test_taxa_with_special_characters
    res = app(self.tmp_seq_filepath)

  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/assign_taxonomy.py", line 425, in __call__
    max_memory=max_memory, tmp_dir=tmp_dir)
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/pycogent_backports/rdp_classifier.py", line 517, in train_rdp_classifier_and_assign_taxonomy
    rmtree(training_dir)
  File "/nv/hp10/woverholt3/qiime1.6/python-2.7.3-release/lib/python2.7/shutil.py", line 254, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/nv/hp10/woverholt3/qiime1.6/python-2.7.3-release/lib/python2.7/shutil.py", line 252, in rmtree
    os.rmdir(path)

OSError: [Errno 39] Directory not empty: '/nv/hp10/woverholt3/data/qiime_files/tmp/RdpTrainer_Mcfz8b'

======================================================================
ERROR: test_train_on_the_fly (__main__.RdpTaxonAssignerTests)
Training on-the-fly classifies reference sequence correctly with 100% certainty

----------------------------------------------------------------------
Traceback (most recent call last):

  File "tests/test_assign_taxonomy.py", line 677, in test_train_on_the_fly
    obs_assignments = app(self.tmp_seq_filepath)

  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/assign_taxonomy.py", line 425, in __call__
    max_memory=max_memory, tmp_dir=tmp_dir)
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/pycogent_backports/rdp_classifier.py", line 517, in train_rdp_classifier_and_assign_taxonomy
    rmtree(training_dir)
  File "/nv/hp10/woverholt3/qiime1.6/python-2.7.3-release/lib/python2.7/shutil.py", line 254, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/nv/hp10/woverholt3/qiime1.6/python-2.7.3-release/lib/python2.7/shutil.py", line 252, in rmtree
    os.rmdir(path)

OSError: [Errno 39] Directory not empty: '/nv/hp10/woverholt3/data/qiime_files/tmp/RdpTrainer_yocp2i'

======================================================================
ERROR: test_train_on_the_fly_low_memory (__main__.RdpTaxonAssignerTests)
Training on-the-fly with lower heap size classifies reference sequence correctly with 100% certainty

----------------------------------------------------------------------
Traceback (most recent call last):

  File "tests/test_assign_taxonomy.py", line 718, in test_train_on_the_fly_low_memory
    obs_assignments = app(self.tmp_seq_filepath)

  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/assign_taxonomy.py", line 425, in __call__
    max_memory=max_memory, tmp_dir=tmp_dir)
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/pycogent_backports/rdp_classifier.py", line 517, in train_rdp_classifier_and_assign_taxonomy
    rmtree(training_dir)
  File "/nv/hp10/woverholt3/qiime1.6/python-2.7.3-release/lib/python2.7/shutil.py", line 254, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/nv/hp10/woverholt3/qiime1.6/python-2.7.3-release/lib/python2.7/shutil.py", line 252, in rmtree
    os.rmdir(path)

OSError: [Errno 39] Directory not empty: '/nv/hp10/woverholt3/data/qiime_files/tmp/RdpTrainer_JjNnN9'

======================================================================
ERROR: test_call_logs_run (__main__.RtaxTaxonAssignerTests)
RtaxTaxonAssigner.__call__ logs the run when expected

----------------------------------------------------------------------
Traceback (most recent call last):

  File "tests/test_assign_taxonomy.py", line 492, in test_call_logs_run
    actual = p(self.input_seqs_fp,log_path=log_path)
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/assign_taxonomy.py", line 742, in __call__
    log_path=log_path,base_tmp_dir=get_qiime_temp_dir())
  File "/nv/hp10/woverholt3/qiime1.6/pycogent-1.5.3-release/lib/python2.7/site-packages/cogent/app/rtax.py", line 163, in assign_taxonomy
    % usearch_command
ApplicationNotFoundError: Cannot find usearch. Is it installed? Is it in your path?

======================================================================
ERROR: test_call_output_to_file (__main__.RtaxTaxonAssignerTests)
RtaxTaxonAssigner.__call__ functions w output to file

----------------------------------------------------------------------
Traceback (most recent call last):

  File "tests/test_assign_taxonomy.py", line 466, in test_call_output_to_file
    actual = p(self.input_seqs_fp, result_path=result_path)
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/assign_taxonomy.py", line 742, in __call__
    log_path=log_path,base_tmp_dir=get_qiime_temp_dir())
  File "/nv/hp10/woverholt3/qiime1.6/pycogent-1.5.3-release/lib/python2.7/site-packages/cogent/app/rtax.py", line 163, in assign_taxonomy
    % usearch_command
ApplicationNotFoundError: Cannot find usearch. Is it installed? Is it in your path?

======================================================================
ERROR: test_call_paired_result (__main__.RtaxTaxonAssignerTests)

----------------------------------------------------------------------
Traceback (most recent call last):

  File "tests/test_assign_taxonomy.py", line 449, in test_call_paired_result
    actual = p(self.input_seqs_fp)
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/assign_taxonomy.py", line 742, in __call__
    log_path=log_path,base_tmp_dir=get_qiime_temp_dir())
  File "/nv/hp10/woverholt3/qiime1.6/pycogent-1.5.3-release/lib/python2.7/site-packages/cogent/app/rtax.py", line 163, in assign_taxonomy
    % usearch_command
ApplicationNotFoundError: Cannot find usearch. Is it installed? Is it in your path?

======================================================================
ERROR: test_call_paired_with_fallback_result (__main__.RtaxTaxonAssignerTests)

----------------------------------------------------------------------
Traceback (most recent call last):

  File "tests/test_assign_taxonomy.py", line 436, in test_call_paired_with_fallback_result
    actual = p(self.input_seqs_fp)
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/assign_taxonomy.py", line 742, in __call__
    log_path=log_path,base_tmp_dir=get_qiime_temp_dir())
  File "/nv/hp10/woverholt3/qiime1.6/pycogent-1.5.3-release/lib/python2.7/site-packages/cogent/app/rtax.py", line 163, in assign_taxonomy
    % usearch_command
ApplicationNotFoundError: Cannot find usearch. Is it installed? Is it in your path?

======================================================================
ERROR: test_call_single_result (__main__.RtaxTaxonAssignerTests)

----------------------------------------------------------------------
Traceback (most recent call last):

  File "tests/test_assign_taxonomy.py", line 422, in test_call_single_result
    actual = p(self.input_seqs_fp)
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/assign_taxonomy.py", line 742, in __call__
    log_path=log_path,base_tmp_dir=get_qiime_temp_dir())
  File "/nv/hp10/woverholt3/qiime1.6/pycogent-1.5.3-release/lib/python2.7/site-packages/cogent/app/rtax.py", line 163, in assign_taxonomy
    % usearch_command
ApplicationNotFoundError: Cannot find usearch. Is it installed? Is it in your path?

----------------------------------------------------------------------
Ran 44 tests in 41.857s

FAILED (errors=11)
----------------------------------------------------------------------

Output from test_parallel/test_assign_taxonomy.py

ERROR: test_parallel_rdp_taxonomy_assigner (__main__.ParallelRdpTaxonomyAssignerTests)
parallel_rdp_taxonomy_assigner functions as expected

----------------------------------------------------------------------
Traceback (most recent call last):

  File "tests/test_parallel/test_assign_taxonomy.py", line 101, in test_parallel_rdp_taxonomy_assigner
    suppress_submit_jobs=False)
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/parallel/util.py", line 189, in __call__
    stdout, stderr, return_value = qiime_system_call(poller_command)
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/util.py", line 1114, in qiime_system_call
    stdout, stderr = proc.communicate()
  File "/nv/hp10/woverholt3/qiime1.6/python-2.7.3-release/lib/python2.7/subprocess.py", line 754, in communicate
    return self._communicate(input)
  File "/nv/hp10/woverholt3/qiime1.6/python-2.7.3-release/lib/python2.7/subprocess.py", line 1312, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/nv/hp10/woverholt3/qiime1.6/python-2.7.3-release/lib/python2.7/subprocess.py", line 1366, in _communicate_with_poll
    ready = poller.poll()
  File "/nv/hp10/woverholt3/qiime1.6/qiime-1.6.0-release/lib/qiime/test.py", line 41, in timeout
    "Test failed to run in allowed time (%d seconds)." % seconds
TimeExceededError: Test failed to run in allowed time (60 seconds).

----------------------------------------------------------------------
Ran 2 tests in 64.943s

FAILED (errors=1)

---------------------------------------------------------------------

Let me know if you need anything else and I hope you have a happy New Year! Thanks again!

Best Regards,
Will

[Kyle Bittinger]

Will,

Thanks for running the tests.  I am going to make some changes in the code and issue a fix for you by Thursday.

Best,

Kyle

--
 
 
 

[Kyle Bittinger]

Will,

I am just waiting for my changes to be pulled into the main QIIME repository, and then I will send you instructions to work around the problem.

--Kyle

[Will Overholt]

Excellent, thanks for the update and all the help with this!
Is it possibly to briefly explain why that error was being called in our system?

Thanks again!

Best Regards,
Will

[Kyle Bittinger]

I still do not know.  With the update, your system should output a warning message and continue with its work, instead of halting on the error.  The new warning message will potentially help diagnose the problem.  My ultimate preference is to completely fix the error -- hopefully you'll be nice enough to keep sending me info after you've applied the update!

--
 
 
 

[David Soong]

Hi

The problem is in lib/qiime/pycogent_backports/rdp_classifier.py: when you rmtree(training_dir) in train_rdp_classifier_and_assign_taxonomy , some of the file handles in training_dir haven't been released by training_results. You can delete training_results to free up the file handles and rmtree would work.

i.e. 

    if model_output_dir is None:

        del training_results

        rmtree(training_dir)

==

David Soong, PhD

Investigator 

GlaxoSmithKline

[Jai Ram Rideout]

Thanks for suggesting a way to fix this, David! For reference, I've created an issue on our issue tracker here:

https://github.com/qiime/qiime/issues/674

Best,

Jai

--
 
---
You received this message because you are subscribed to the Google Groups "Qiime Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qiime-forum...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[David Soong]

Glad I could help!
David