Qiime 1.9.1 Uclust Fatal error pick_de_novo_otu.py

[azadeh.sa...@gmail.com]

Hi 

I have used pick_de_novo_otu  on VMBox since 2015, and recently I have a strange error,

I checked several parameters , and every thing looks me fine,

I tested pick_de_novo_otu on a test data file containing 5 samples, it works fine, finishes fine, and had no error

Here the error come out with pick_de_novo_otu on my original data (96 sample 3.6 Giga ) :

Logging started at 02:45:34 on 27 Apr 2018

QIIME version: 1.9.1

qiime_config values:

blastmat_dir /qiime_software/blast-2.2.22-release/data

pick_otus_reference_seqs_fp /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

sc_queue all.q

pynast_template_alignment_fp /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.pynast.fasta

cluster_jobs_fp start_parallel_jobs.py

assign_taxonomy_reference_seqs_fp /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

torque_queue friendlyq

jobs_to_start 1

denoiser_min_per_core 50

assign_taxonomy_id_to_taxonomy_fp /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt

temp_dir /tmp/

blastall_fp /qiime_software/blast-2.2.22-release/bin/blastall

seconds_to_sleep 1

parameter file values:

parallel:jobs_to_start 1

Input file md5 sums:

split_librairies_hmn_four/seqs.fna: 0175e1a16da961a99414fcc4474e3065

Executing commands.

# Pick OTUs command 

pick_otus.py -i split_librairies_hmn_four/seqs.fna -o pick_de_novo_otus_greengene_secondtry/uclust_picked_otus 

Stdout:

Stderr:

# Pick representative set command 

pick_rep_set.py -i pick_de_novo_otus_greengene_secondtry/uclust_picked_otus/seqs_otus.txt -f split_librairies_hmn_four/seqs.fna -l pick_de_novo_otus_greengene_secondtry/rep_set//seqs_rep_set.log -o pick_de_novo_otus_greengene_secondtry/rep_set//seqs_rep_set.fasta 

Stdout:

Stderr:

# Assign taxonomy command 

assign_taxonomy.py -o pick_de_novo_otus_greengene_secondtry/uclust_assigned_taxonomy -i pick_de_novo_otus_greengene_secondtry/rep_set//seqs_rep_set.fasta 

*** ERROR RAISED DURING STEP: Assign taxonomy

Command run was:

 assign_taxonomy.py -o pick_de_novo_otus_greengene_secondtry/uclust_assigned_taxonomy -i pick_de_novo_otus_greengene_secondtry/rep_set//seqs_rep_set.fasta 

Command returned exit status: 1

Stdout:

Stderr

Traceback (most recent call last):

  File "/usr/local/bin/assign_taxonomy.py", line 417, in <module>

    main()

  File "/usr/local/bin/assign_taxonomy.py", line 394, in main

    log_path=log_path)

  File "/usr/local/lib/python2.7/dist-packages/qiime/assign_taxonomy.py", line 1304, in __call__

    '--uc': uc_path})

  File "/usr/local/lib/python2.7/dist-packages/burrito/util.py", line 285, in __call__

    'StdErr:\n%s\n' % open(errfile).read())

burrito.util.ApplicationError: Unacceptable application exit status: 1

Command:

cd "/home/qiime/workspace/16S_HMN_FOURTH_RUN/"; uclust --input "pick_de_novo_otus_greengene_secondtry/rep_set//seqs_rep_set.fasta" --id 0.9 --rev --maxaccepts 3 --allhits --libonly --lib "/usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta" --uc "/tmp/UclustConsensusTaxonAssigner_xMXAMt.uc" > "/tmp/tmpNAfCOVETkH1QyzU6y0dy.txt" 2> "/tmp/tmptQ03uXxUulpsG6u2fcEx.txt"

StdOut:

StdErr:

uclust v1.2.22q

(C) Copyright 2009-10 Robert C. Edgar

Licensed ONLY for use in PyNAST and QIIME.

00:00  37Mb    0.0% Reading lib, 0 seeds

00:01  37Mb    0.0% Reading lib, 1 seeds

00:02 177Mb    3.8% Reading lib, 3798 seeds

00:03 398Mb    9.7% Reading lib, 9638 seeds

00:04 578Mb   14.4% Reading lib, 14360 seeds

00:05 849Mb   21.6% Reading lib, 21552 seeds

00:06 1.0Gb   26.5% Reading lib, 26468 seeds

00:07 1.3Gb   33.3% Reading lib, 33268 seeds

00:08 1.5Gb   39.0% Reading lib, 38896 seeds

00:09 1.8Gb   45.5% Reading lib, 45533 seeds

00:10 1.9Gb   50.4% Reading lib, 50397 seeds

00:11 2.2Gb   57.1% Reading lib, 56917 seeds

00:12 2.4Gb   62.2% Reading lib, 61985 seeds

00:13 2.6Gb   68.4% Reading lib, 68118 seeds

00:14 2.8Gb   73.6% Reading lib, 73242 seeds

00:15 3.0Gb   79.8% Reading lib, 79323 seeds

00:16 3.2Gb   84.3% Reading lib, 83787 seeds

00:17 3.4Gb   89.0% Reading lib, 88372 seeds

00:18 3.6Gb   93.8% Reading lib, 93101 seeds

00:19 3.8Gb   99.0% Reading lib, 98264 seeds

00:19 3.8Gb  100.0% Reading lib, 99322 seeds

00:19 3.8Gb    0.0%                         

00:20 3.8Gb    0.0% 78.4% matched to lib at 90.0%, id 96.7%

00:21 3.8Gb    0.1% 77.2% matched to lib at 90.0%, id 96.8%

00:23 3.8Gb    0.2% 75.7% matched to lib at 90.0%, id 97.0%

00:25 3.8Gb    0.3% 75.4% matched to lib at 90.0%, id 97.1%

00:26 3.8Gb    0.3% 74.9% matched to lib at 90.0%, id 97.1%

00:27 3.8Gb    0.4% 74.6% matched to lib at 90.0%, id 97.0%

00:28 3.8Gb    0.5% 74.2% matched to lib at 90.0%, id 97.0%

00:29 3.8Gb    0.5% 74.6% matched to lib at 90.0%, id 97.0%

00:30 3.8Gb    0.5% 74.1% matched to lib at 90.0%, id 97.0%

00:33 3.8Gb    0.7% 73.8% matched to lib at 90.0%, id 97.0%

00:34 3.8Gb    0.7% 73.7% matched to lib at 90.0%, id 97.0%

00:35 3.8Gb    0.7% 73.7% matched to lib at 90.0%, id 97.0%

00:36 3.8Gb    0.8% 73.5% matched to lib at 90.0%, id 97.0%

00:37 3.8Gb    0.8% 73.6% matched to lib at 90.0%, id 97.0%

00:38 3.8Gb    0.9% 73.7% matched to lib at 90.0%, id 97.0%

00:39 3.8Gb    0.9% 73.5% matched to lib at 90.0%, id 97.0%

00:40 3.8Gb    1.0% 73.5% matched to lib at 90.0%, id 97.0%

00:41 3.8Gb    1.1% 73.7% matched to lib at 90.0%, id 97.0%

00:42 3.8Gb    1.1% 73.4% matched to lib at 90.0%, id 97.0%

00:44 3.8Gb    1.2% 73.4% matched to lib at 90.0%, id 97.0%

00:45 3.8Gb    1.2% 73.7% matched to lib at 90.0%, id 97.0%

00:48 3.8Gb    1.3% 73.4% matched to lib at 90.0%, id 97.0%

00:49 3.8Gb    1.4% 73.5% matched to lib at 90.0%, id 97.0%

00:52 3.8Gb    1.5% 73.8% matched to lib at 90.0%, id 97.0%

00:53 3.8Gb    1.6% 73.8% matched to lib at 90.0%, id 97.0%

00:54 3.8Gb    1.6% 73.8% matched to lib at 90.0%, id 97.0%

00:55 3.8Gb    1.7% 73.7% matched to lib at 90.0%, id 97.0%

00:57 3.8Gb    1.8% 73.5% matched to lib at 90.0%, id 97.0%

00:58 3.8Gb    1.8% 73.5% matched to lib at 90.0%, id 97.0%

01:01 3.8Gb    2.0% 73.5% matched to lib at 90.0%, id 97.0%

01:02 3.8Gb    2.0% 73.5% matched to lib at 90.0%, id 97.0%

01:03 3.8Gb    2.0% 73.5% matched to lib at 90.0%, id 97.0%

01:06 3.8Gb    2.2% 73.6% matched to lib at 90.0%, id 97.0%

01:07 3.8Gb    2.2% 73.6% matched to lib at 90.0%, id 97.0%

01:08 3.8Gb    2.2% 73.5% matched to lib at 90.0%, id 97.0%

01:09 3.8Gb    2.3% 73.3% matched to lib at 90.0%, id 97.0%

01:10 3.8Gb    2.3% 73.4% matched to lib at 90.0%, id 97.0%

01:11 3.8Gb    2.4% 73.4% matched to lib at 90.0%, id 97.0%

01:12 3.8Gb    2.4% 73.3% matched to lib at 90.0%, id 97.0%

01:13 3.8Gb    2.4% 73.3% matched to lib at 90.0%, id 97.1%

01:16 3.8Gb    2.6% 73.2% matched to lib at 90.0%, id 97.1%

01:17 3.8Gb    2.6% 73.1% matched to lib at 90.0%, id 97.1%

01:21 3.8Gb    2.8% 73.1% matched to lib at 90.0%, id 97.1%

01:24 3.8Gb    2.9% 73.0% matched to lib at 90.0%, id 97.0%

01:25 3.8Gb    3.0% 73.0% matched to lib at 90.0%, id 97.0%

01:26 3.8Gb    3.0% 73.1% matched to lib at 90.0%, id 97.1%

01:30 3.8Gb    3.2% 72.9% matched to lib at 90.0%, id 97.1%

29:30 3.8Gb   78.4% 72.7% matched to lib at 90.0%, id 97.1%

29:31 3.8Gb   78.4% 72.7% matched to lib at 90.0%, id 97.1%

29:32 3.8Gb   78.4% 72.7% matched to lib at 90.0%, id 97.1%

29:35 3.8Gb   78.6% 72.7% matched to lib at 90.0%, id 97.1%

29:36 3.8Gb   78.6% 72.7% matched to lib at 90.0%, id 97.1%

29:40 3.8Gb   78.8% 72.7% matched to lib at 90.0%, id 97.1%

29:42 3.8Gb   78.9% 72.7% matched to lib at 90.0%, id 97.1%

29:44 3.8Gb   79.0% 72.7% matched to lib at 90.0%, id 97.1%

29:50 3.8Gb   79.3% 72.7% matched to lib at 90.0%, id 97.1%

29:51 3.8Gb   79.3% 72.7% matched to lib at 90.0%, id 97.1%

29:52 3.8Gb   79.4% 72.7% matched to lib at 90.0%, id 97.1%

29:54 3.8Gb   79.4% 72.7% matched to lib at 90.0%, id 97.1%

29:56 3.8Gb   79.5% 72.7% matched to lib at 90.0%, id 97.1%

29:58 3.8Gb   79.6% 72.7% matched to lib at 90.0%, id 97.1%

29:59 3.8Gb   79.6% 72.7% matched to lib at 90.0%, id 97.1%

30:00 3.8Gb   79.7% 72.7% matched to lib at 90.0%, id 97.1%

---Fatal error---

tracebackbit.cpp(64) assert failed: i > 0 && j > 0

uclust --input pick_de_novo_otus_greengene_secondtry/rep_set//seqs_rep_set.fasta --id 0.9 --rev --maxaccepts 3 --allhits --libonly --lib /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta --uc /tmp/UclustConsensusTaxonAssigner_xMXAMt.uc

Elapsed time: 30:00

Here Uclust parameters :

UclustOtuPicker parameters:

Application:uclust

Similarity:0.97

enable_rev_strand_matching:False

exact:False

max_accepts:1

max_rejects:8

new_cluster_identifier:denovo

optimal:False

output_dir:pick_de_novo_otus_greengene_secondtry/uclust_picked_otus

prefilter_identical_sequences:True

presort_by_abundance:True

save_uc_files:True

stable_sort:True

stepwords:8

suppress_sort:True

word_length:8

Num OTUs:277224

Result path: pick_de_novo_otus_greengene_secondtry/uclust_picked_otus/seqs_otus.txt

and then qiime configuration parameters:

qiime@qiime-190-virtual-box:~/workspace/16S_HMN_FOURTH_RUN$ print_qiime_config.py -ft

System information

==================

         Platform: linux2

   Python version: 2.7.3 (default, Jun 22 2015, 19:33:41)  [GCC 4.6.3]

Python executable: /usr/bin/python

QIIME default reference information

===================================

For details on what files are used as QIIME's default references, see here:

 https://github.com/biocore/qiime-default-reference/releases/tag/0.1.2

Dependency versions

===================

                QIIME library version: 1.9.1

                 QIIME script version: 1.9.1

      qiime-default-reference version: 0.1.2

                        NumPy version: 1.9.2

                        SciPy version: 0.15.1

                       pandas version: 0.16.1

                   matplotlib version: 1.4.3

                  biom-format version: 2.1.4

                         h5py version: 2.4.0 (HDF5 version: 1.8.4)

                         qcli version: 0.1.1

                         pyqi version: 0.3.2

                   scikit-bio version: 0.2.3

                       PyNAST version: 1.2.2

                      Emperor version: 0.9.51

                      burrito version: 0.9.1

             burrito-fillings version: 0.1.1

                    sortmerna version: SortMeRNA version 2.0, 29/11/2014

                    sumaclust version: SUMACLUST Version 1.0.00

                        swarm version: Swarm 1.2.19 [May 26 2015 13:50:14]

                                gdata: Installed.

RDP Classifier version (if installed): rdp_classifier-2.2.jar

          Java version (if installed): 1.6.0_39

QIIME config values

===================

For definitions of these settings and to learn how to configure QIIME, see here:

 http://qiime.org/install/qiime_config.html

 http://qiime.org/tutorials/parallel_qiime.html

                     blastmat_dir: /qiime_software/blast-2.2.22-release/data

      pick_otus_reference_seqs_fp: /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

                         sc_queue: all.q

      topiaryexplorer_project_dir: None

     pynast_template_alignment_fp: /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.pynast.fasta

                  cluster_jobs_fp: start_parallel_jobs.py

pynast_template_alignment_blastdb: None

assign_taxonomy_reference_seqs_fp: /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

                     torque_queue: friendlyq

                    jobs_to_start: 1

                       slurm_time: None

            denoiser_min_per_core: 50

assign_taxonomy_id_to_taxonomy_fp: /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt

                         temp_dir: /tmp/

                     slurm_memory: None

                      slurm_queue: None

                      blastall_fp: /qiime_software/blast-2.2.22-release/bin/blastall

                 seconds_to_sleep: 1

QIIME full install test results

===============================

..........................F

======================================================================

FAIL: test_usearch_supported_version (__main__.QIIMEDependencyFull)

usearch is in path and version is supported

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/bin/print_qiime_config.py", line 650, in test_usearch_supported_version

    "which components of QIIME you plan to use.")

AssertionError: usearch not found. This may or may not be a problem depending on which components of QIIME you plan to use.

----------------------------------------------------------------------

Ran 27 tests in 0.095s

FAILED (failures=1)

I gave 25G ram to VM and . 

Since I performed the same analysis for different groups of samples, I took an old dataset for which I had actually ran pick_de_novo_otu without any problem, and try to run pick_de_novo_otu on it again, the same error that I mentioned above came up. !!! 

Another strange issue happened, I was struggeling to get executed pick_de_novo_otus, I realized that Shared_Folder was disapeared, 

(which was shared between  MAC and VMbox) I have no explanation why that was happened, and I created another Shared_Folder.

Now I don't know what to do, 

Can you help me ?

Thank you in advance,

Azadeh 

 

[Colin Brislawn]

Hello Azadeh,

Thanks for posting your detailed log files and your VM setup. This is very strange! I'm not totally sure what's going on...

30:00 3.8Gb   79.7% 72.7% matched to lib at 90.0%, id 97.1%

That line makes me think your VM is running out of RAM, which is strange because having 25GB is plenty of memory allocated. Could you check that the VM didn't reset your settings? If the VM is messing up the Shared Folder and other settings, maybe RAM got messed up too.

Let me know what you find. If it's not RAM, we could try testing other things,

Colin

[azadeh.sa...@gmail.com]

Hello Colin

Thanks for your answer,

I have still the same problem with Qiime,

Here you see, 

I asked the amount of ram ( free -g, top) , while nothing was running I copied the answer here,   and it seems there is  24G ram, that VM identifies it,

(but I don't understand  why the free ram is only 10G  it doesn't make sens?? ) 

qiime@qiime-190-virtual-box:~/workspace$ free -g

             total       used       free     shared    buffers     cached

Mem:            24         13         10          0          0         12

-/+ buffers/cache:          0         23

Swap:            1          0          1

qiime@qiime-190-virtual-box:~/workspace$ top

top - 01:13:55 up 4 days, 22:31,  3 users,  load average: 0.00, 0.01, 0.05

Tasks: 154 total,   2 running, 152 sleeping,   0 stopped,   0 zombie

Cpu(s):  0.3%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:  25299616k total, 13852200k used, 11447416k free,   177308k buffers

Swap:  2095100k total,        0k used,  2095100k free, 12695180k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                        

 1158 root      20   0  301m 134m  23m S  0.3  0.5   0:46.37 Xorg                                                                                           

 1677 qiime     20   0 1063m  95m  36m S  0.3  0.4   5:17.42 unity-2d-shell                                                                                 

 1991 qiime     20   0  515m  24m  11m S  0.3  0.1   0:13.89 gnome-terminal                                                                                 

    1 root      20   0 24432 2420 1348 S  0.0  0.0   0:00.84 init                                                                                           

    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd                                                                                       

    3 root      20   0     0    0    0 S  0.0  0.0   0:00.94 ksoftirqd/0

Do you have any idea what is using the ram ? 

Shall I reset the VM ?

I can reconfigure it ..make a new VM with Qiime, do you think that could solve the issue ?

Thank you again,

Azadeh 

[azadeh.sa...@gmail.com]

Hello again

After restarting the VM, here what I have:

qiime@qiime-190-virtual-box:~$ free -g

             total       used       free     shared    buffers     cached

Mem:            24          0         23          0          0          0

-/+ buffers/cache:          0         23

Swap:            1          0          1

Used and Cached memory are zero.

and 24 giga total ram.

So there is enough ram for qiime.

[Colin Brislawn]

I'm glad you got it working! Yes, restarting the VM should reduce the memory used.

Were you able to run the scripts you needed?

Colin

[azadeh.sa...@gmail.com]

Hello

No, Actually I have the same error, running pick-de-novo-otu on VM with 24g ram produces the same error that I mentionned priviously.
Shall I reset VM ?

[Colin Brislawn]

Ah ok. 

Did you get the same uclust error with something like

3.8Gb   79.7% 72.7% matched to lib at 90.0%, id 97.1%

I'm wondering if there is some issue with uclust and total memory allowed.

Can you run 

which uclust

in the linux terminal?

Colin

[azadeh.sa...@gmail.com]

Hello

Thank you for your answer

Yes,  again, the same error :

qiime@qiime-190-virtual-box:~/workspace/16S_HMN_FOURTH_RUN$ pick_de_novo_otus.py -i split_librairies_hmn_four/seqs.fna -o all_otu_gg_after_restarting

Traceback (most recent call last):

  File "/usr/local/bin/pick_de_novo_otus.py", line 180, in <module>

    main()

  File "/usr/local/bin/pick_de_novo_otus.py", line 177, in main

    status_update_callback=status_update_callback)

  File "/usr/local/lib/python2.7/dist-packages/qiime/workflow/upstream.py", line 306, in run_pick_de_novo_otus

    close_logger_on_success=close_logger_on_success)

  File "/usr/local/lib/python2.7/dist-packages/qiime/workflow/util.py", line 122, in call_commands_serially

    raise WorkflowError(msg)

qiime.workflow.util.WorkflowError: 

*** ERROR RAISED DURING STEP: Assign taxonomy

Command run was:

 assign_taxonomy.py -o all_otu_gg_after_restarting/uclust_assigned_taxonomy -i all_otu_gg_after_restarting/rep_set//seqs_rep_set.fasta 

Command returned exit status: 1

Stdout:

Stderr

Traceback (most recent call last):

  File "/usr/local/bin/assign_taxonomy.py", line 417, in <module>

    main()

  File "/usr/local/bin/assign_taxonomy.py", line 394, in main

    log_path=log_path)

  File "/usr/local/lib/python2.7/dist-packages/qiime/assign_taxonomy.py", line 1304, in __call__

    '--uc': uc_path})

  File "/usr/local/lib/python2.7/dist-packages/burrito/util.py", line 285, in __call__

    'StdErr:\n%s\n' % open(errfile).read())

burrito.util.ApplicationError: Unacceptable application exit status: 1

Command:

cd "/home/qiime/workspace/16S_HMN_FOURTH_RUN/"; uclust --input "all_otu_gg_after_restarting/rep_set//seqs_rep_set.fasta" --id 0.9 --rev --maxaccepts 3 --allhits --libonly --lib "/usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta" --uc "/tmp/UclustConsensusTaxonAssigner_YiRVIX.uc" > "/tmp/tmpHCoV6hOWzqd5R4ZymiT6.txt" 2> "/tmp/tmpibywQ9kiG1lM9ZVZxmoU.txt"

StdOut:

StdErr:

uclust v1.2.22q

(C) Copyright 2009-10 Robert C. Edgar

Licensed ONLY for use in PyNAST and QIIME.

00:00  37Mb    0.0% Reading lib, 0 seeds

00:01 245Mb    5.6% Reading lib, 5598 seeds

00:02 479Mb   11.8% Reading lib, 11746 seeds

00:03 752Mb   19.0% Reading lib, 18935 seeds

00:04 936Mb   24.0% Reading lib, 23970 seeds

00:05 1.2Gb   31.2% Reading lib, 31169 seeds

00:06 1.4Gb   37.2% Reading lib, 37112 seeds

00:07 1.7Gb   44.2% Reading lib, 44216 seeds

00:08 1.9Gb   50.3% Reading lib, 50244 seeds

00:09 2.2Gb   56.6% Reading lib, 56484 seeds

00:10 2.4Gb   62.2% Reading lib, 61985 seeds

00:11 2.6Gb   68.4% Reading lib, 68118 seeds

00:12 2.8Gb   73.5% Reading lib, 73127 seeds

00:13 3.0Gb   79.4% Reading lib, 78998 seeds

00:14 3.3Gb   84.9% Reading lib, 84419 seeds

00:15 3.5Gb   90.7% Reading lib, 90058 seeds

00:16 3.7Gb   95.9% Reading lib, 95227 seeds

00:16 3.8Gb  100.0% Reading lib, 99322 seeds

00:16 3.8Gb    0.0%                         

00:17 3.8Gb    0.0% 77.6% matched to lib at 90.0%, id 96.7%

00:18 3.8Gb    0.1% 78.8% matched to lib at 90.0%, id 96.8%

00:19 3.8Gb    0.1% 77.2% matched to lib at 90.0%, id 96.8%

00:20 3.8Gb    0.2% 75.4% matched to lib at 90.0%, id 97.0%

00:23 3.8Gb    0.3% 75.1% matched to lib at 90.0%, id 97.1%

00:24 3.8Gb    0.4% 74.8% matched to lib at 90.0%, id 97.1%

.

.

.

.

2:52 3.8Gb    6.9% 73.1% matched to lib at 90.0%, id 97.1%

02:53 3.8Gb    7.0% 73.0% matched to lib at 90.0%, id 97.1%

02:54 3.8Gb    7.0% 73.1% matched to lib at 90.0%, id 97.1%

---Fatal error---

tracebackbit.cpp(64) assert failed: i > 0 && j > 0

uclust --input all_otu_gg_after_restarting/rep_set//seqs_rep_set.fasta --id 0.9 --rev --maxaccepts 3 --allhits --libonly --lib /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta --uc /tmp/UclustConsensusTaxonAssigner_YiRVIX.uc

Elapsed time: 02:55

Here is uclust location: 

qiime@qiime-190-virtual-box:~/workspace/16S_HMN_FOURTH_RUN$ which uclust

/usr/local/bin/uclust

Other point, I watched the monitor of activity ( which is an application of IMAC),  I watched the amount of  ram used by VM while Qiime was running, it gets never more than 14.6G , 

It gets stable on 14.6G and then uclust crashes, ...  

The amount of ram allocated to VM was 16G (from the first day of using Qiime on the VM until some months ago) , and it was fine, I used Qiime on the VM for several projects and I've used pick_de_novo_otu  every time without any kind of error.

 For only one projet pynast didn't run well in the level of making alignement for phylogenetic tree,  giving an error of "memory not enough" ( more than 1000 000 otu was identified) , then I added more ram, allocated 24G  (from  total  ram of 32G of my Imac). Unfortunately it didn't work for that project,  crashed giving the same error "memory not enough". 

Afterward qiime didn't work, sending the same error message  "00:17 3.8Gb    0.0% 77.6% matched to lib at 90.0%, id 96.7%" . Even-though I tried using the old data for which I had run pick_de_novo_otu and had obtained result (before changing the amount of allocated 16G ram ), it didn't work uclust gave the same error  that I posted above.

Now I'm really confused and I need to work it out. Maybe it's easier to setup a new VM. 

Thank you for your help,

Have a nice day

Azadeh 

    

 

  

[Colin Brislawn]

Hello Azadeh,

Thanks for posting all of this information. I think we are on the right path by investigating RAM / memory. Having 24GB allocated to the VM should be enough, which is why it's strange that only 14.6GB is being used, and why uclust only shows 3.8GB when it crashes. 

Setting up a new VM is a good way to troubleshoot with this. Let me know if that works.

If you are running this on a Mac (OSX), you could install qiime natively without the VM. 

Colin

[azadeh.sa...@gmail.com]

Hello Colin

Thank you for your answer,

By installing Qiime directly on my IMAC I had a problem for running uclust and usearch, ( licence problem)

I tried to run Qiime directly on my IMAC, couldn't make work uclust properly, decided to use Qiime through a VM.

Thank you again,

Regards,

Azadeh

[Colin Brislawn]

Hello Azadeh,

I've attached the 'uclust' program that works on my mac. This is the official executable that was licensed just for Qiime 1 and Pynast.

Qiime should work inside the VM as well, especially if you have enough RAM on the VM.

Colin

[azadeh.sa...@gmail.com]

Hello Colin

Thank you for your answer,

I come back to you, 

I actually resolved the last problem, I setup VM completely. 

And then  pick_de_novo_otu worked very well, It was running for a while and  finished properly.

But following the analysis by using core_diversity_analysis arise a segmentation_fault error  : ( very strange)

Logging started at 08:33:38 on 14 May 2018

QIIME version: 1.9.1

qiime_config values:

blastmat_dir /qiime_software/blast-2.2.22-release/data

pick_otus_reference_seqs_fp /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

sc_queue all.q

pynast_template_alignment_fp /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.pynast.fasta

cluster_jobs_fp start_parallel_jobs.py

assign_taxonomy_reference_seqs_fp /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

torque_queue friendlyq

jobs_to_start 1

denoiser_min_per_core 50

assign_taxonomy_id_to_taxonomy_fp /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt

temp_dir /tmp/

blastall_fp /qiime_software/blast-2.2.22-release/bin/blastall

seconds_to_sleep 1

parameter file values:

parallel:jobs_to_start 1

Input file md5 sums:

ALL_SAMPS_OTU_DE_NOVO/otu_table_NoUnassigned.biom: 402474aadfb1ffbbf1236d14e2fc92d9

mapping_table_HMN4.txt: f1bd778ee76c26703df236dfe957a7a2

ALL_SAMPS_OTU_DE_NOVO/rep_set.tre: a974540041b1aee88b7867fc3846fcad

Executing commands.

# Generate BIOM table summary command 

biom summarize-table -i ALL_SAMPS_OTU_DE_NOVO/otu_table_NoUnassigned.biom -o CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/biom_table_summary.txt 

Stdout:

Stderr:

# Filter low sequence count samples from table (minimum sequence count: 20) command 

filter_samples_from_otu_table.py -i ALL_SAMPS_OTU_DE_NOVO/otu_table_NoUnassigned.biom -o CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/table_mc20.biom -n 20

Stdout:

Stderr:

# Rarify the OTU table to 20 sequences/sample command 

single_rarefaction.py -i CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/table_mc20.biom -o CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/table_even20.biom -d 20

Stdout:

Stderr:

Executing commands.

# Beta Diversity (weighted_unifrac) command 

beta_diversity.py -i CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/table_even20.biom -o CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/bdiv_even20/ --metrics weighted_unifrac  -t ALL_SAMPS_OTU_DE_NOVO/rep_set.tre 

Stdout:

Stderr:

# Rename distance matrix (weighted_unifrac) command 

mv CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/bdiv_even20//weighted_unifrac_table_even20.txt CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/bdiv_even20//weighted_unifrac_dm.txt

Stdout:

Stderr:

# Principal coordinates (weighted_unifrac) command 

principal_coordinates.py -i CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/bdiv_even20//weighted_unifrac_dm.txt -o CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/bdiv_even20//weighted_unifrac_pc.txt 

Stdout:

Stderr:

/usr/local/lib/python2.7/dist-packages/skbio/stats/ordination/_principal_coordinate_analysis.py:107: RuntimeWarning: The result contains negative eigenvalues. Please compare their magnitude with the magnitude of some of the largest positive eigenvalues. If the negative ones are smaller, it's probably safe to ignore them, but if they are large in magnitude, the results won't be useful. See the Notes section for more details. The smallest eigenvalue is -0.154394873363 and the largest is 3.16908879393.

  RuntimeWarning

# Make emperor plots, weighted_unifrac) command 

make_emperor.py -i CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/bdiv_even20//weighted_unifrac_pc.txt -o CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/bdiv_even20//weighted_unifrac_emperor_pcoa_plot/ -m mapping_table_HMN4.txt 

Stdout:

Stderr:

# Beta Diversity (unweighted_unifrac) command 

beta_diversity.py -i CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/table_even20.biom -o CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/bdiv_even20/ --metrics unweighted_unifrac  -t ALL_SAMPS_OTU_DE_NOVO/rep_set.tre 

Stdout:

Stderr:

# Rename distance matrix (unweighted_unifrac) command 

mv CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/bdiv_even20//unweighted_unifrac_table_even20.txt CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/bdiv_even20//unweighted_unifrac_dm.txt

Stdout:

Stderr:

# Principal coordinates (unweighted_unifrac) command 

principal_coordinates.py -i CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/bdiv_even20//unweighted_unifrac_dm.txt -o CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/bdiv_even20//unweighted_unifrac_pc.txt 

Stdout:

Stderr:

/usr/local/lib/python2.7/dist-packages/skbio/stats/ordination/_principal_coordinate_analysis.py:107: RuntimeWarning: The result contains negative eigenvalues. Please compare their magnitude with the magnitude of some of the largest positive eigenvalues. If the negative ones are smaller, it's probably safe to ignore them, but if they are large in magnitude, the results won't be useful. See the Notes section for more details. The smallest eigenvalue is -0.0334097926912 and the largest is 3.67294598643.

  RuntimeWarning

# Make emperor plots, unweighted_unifrac) command 

make_emperor.py -i CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/bdiv_even20//unweighted_unifrac_pc.txt -o CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/bdiv_even20//unweighted_unifrac_emperor_pcoa_plot/ -m mapping_table_HMN4.txt 

Stdout:

Stderr:

Executing commands.

# Alpha rarefaction command 

multiple_rarefactions.py -i CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/table_mc20.biom -m 10 -x 20 -s 1 -o CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/arare_max20//rarefaction/ 

*** ERROR RAISED DURING STEP: Alpha rarefaction

Command run was:

 multiple_rarefactions.py -i CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/table_mc20.biom -m 10 -x 20 -s 1 -o CORE_DIVERSITY_ANALYSES_NoUnassigned_e20/arare_max20//rarefaction/ 

Command returned exit status: 139

Stdout:

Stderr

Segmentation fault (core dumped)

Logging stopped at 08:38:31 on 14 May 2018

=========== 

This is also a new error, I've never had it before, 

Do you have any idea what's going wrong ?

Thank you again in advance,

Azadeh

[Colin Brislawn]

Hello Azadeh,

Strange! I've never seen a segmentation fault from multiple_rarefactions.py before! My guess is that this is caused by your crazy low rarefaction level:

Rarify the OTU table to 20 sequences/sample command 

With only 20 reads per sample, you will only see a tiny fraction of the community. Why pick such a small number? Try using a larger value for --sampling_depth and try again.

The good news is that all the other steps in this pipeline worked as expected! You should be able to see the new emperor plots. But I would still rerun this step using a higher value.

Colin

[azadeh.sa...@gmail.com]

Dear Colin

I come again to you,

Because I have again a lot of problem with uclust,

Since some years ago I use qiime on a virtual machine installed on my IMAC, recently I had some problem and I setup again the VM and reinstalled qiime( I sent several message for asking help to qiime forum and you and your colleague helped me a lot), but still I have strange errors time to time, I have several time "Segmentation Fault".

Since two days ago I try to use  'pick_de_novo_otu', and first time it didn't work  returning a "Segmentation Fault" error, I retried the same command second time didn't work neither and this time It was another error from uclust( just before ending of otu picking every things were crashed), so for coping with all these confusing error I decided to do an package uptodate ( package uptodate suggested by linux of VM since some months ago).     

After finishing I executed again 'pick_de_novo_otu', then I had this new error :

** ERROR RAISED DURING STEP: Pick OTUs

Command run was:

 pick_otus.py -i split_librairies_pampa2/seqs.fna -o all_de_novo_otu/uclust_picked_otus 

Command returned exit status: 1

Stdout:

Stderr

Traceback (most recent call last):

  File "/usr/local/bin/pick_otus.py", line 1004, in <module>

    main()

  File "/usr/local/bin/pick_otus.py", line 792, in main

    result_path=result_path, log_path=log_path, HALT_EXEC=False)

  File "/usr/local/lib/python2.7/dist-packages/qiime/pick_otus.py", line 1286, in __call__

    HALT_EXEC=HALT_EXEC)

  File "/usr/local/lib/python2.7/dist-packages/bfillings/uclust.py", line 585, in get_clusters_from_fasta_filepath

    raise ApplicationError('Error running uclust. Possible causes are '

burrito.util.ApplicationError: Error running uclust. Possible causes are unsupported version (current supported version is v1.2.22) is installed or improperly formatted input file was provided

(log file attached to this email)

I'm really confused, do you have any idea how can I solve this problem ?

Here qiime_config:

ystem information

==================

         Platform: linux2

   Python version: 2.7.3 (default, Oct 26 2016, 21:01:49)  [GCC 4.6.3]

Python executable: /usr/bin/python

QIIME default reference information

===================================

For details on what files are used as QIIME's default references, see here:

 https://github.com/biocore/qiime-default-reference/releases/tag/0.1.2

Dependency versions

===================

          QIIME library version: 1.9.1

           QIIME script version: 1.9.1

qiime-default-reference version: 0.1.2

                  NumPy version: 1.9.2

                  SciPy version: 0.15.1

                 pandas version: 0.16.1

             matplotlib version: 1.4.3

            biom-format version: 2.1.4

                   h5py version: 2.7.1 (HDF5 version: 1.8.4)

                   qcli version: 0.1.1

                   pyqi version: 0.3.2

             scikit-bio version: 0.2.3

                 PyNAST version: 1.2.2

                Emperor version: 0.9.51

                burrito version: 0.9.1

       burrito-fillings version: 0.1.1

              sortmerna version: SortMeRNA version 2.0, 29/11/2014

              sumaclust version: SUMACLUST Version 1.0.00

                  swarm version: Swarm 1.2.19 [May 26 2015 13:50:14]

                          gdata: Installed.

QIIME config values

===================

For definitions of these settings and to learn how to configure QIIME, see here:

 http://qiime.org/install/qiime_config.html

 http://qiime.org/tutorials/parallel_qiime.html

                     blastmat_dir: /qiime_software/blast-2.2.22-release/data

      pick_otus_reference_seqs_fp: /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

                         sc_queue: all.q

      topiaryexplorer_project_dir: None

     pynast_template_alignment_fp: /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.pynast.fasta

                  cluster_jobs_fp: start_parallel_jobs.py

pynast_template_alignment_blastdb: None

assign_taxonomy_reference_seqs_fp: /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

                     torque_queue: friendlyq

                    jobs_to_start: 1

                       slurm_time: None

            denoiser_min_per_core: 50

assign_taxonomy_id_to_taxonomy_fp: /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt

                         temp_dir: /tmp/

                     slurm_memory: None

                      slurm_queue: None

                      blastall_fp: /qiime_software/blast-2.2.22-release/bin/blastall

                 seconds_to_sleep: 1

 

Thank you in advance for your help,

Azadeh

[Colin Brislawn]

Hello Azadeh,

Thanks for reaching out. I'm happy to help.

As you can see from the log files, this is the central error:

burrito.util.ApplicationError: Error running uclust. Possible causes are unsupported version (current supported version is v1.2.22) is installed or improperly formatted input file was provided

 

So we need to check that the right version of uclust is installed by running one of these commands: 

uclust -v

uclust

We also should check that the input file is present and in the right format

head split_librairies_pampa2/seqs.fna

Maybe that seqs.fna file is missing or in the wrong format.

Let me know what you find,

Colin

[azadeh.sa...@gmail.com]

Hello Colin

Here you see: 

qiime@qiime-190-virtual-box:~/workspace/16S_BENJAMIN_2$ uclust --version

uclust v1.2.22q

and the input file is also present and fine :

qiime@qiime-190-virtual-box:~/workspace/16S_BENJAMIN_2$ head ./split_librairies_pampa2/seqs.fna 

>SAM.1_0 M01626:391:000000000-BWTWT:1:1104:28121:14251 1:N:0:73 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0

TGGGGAATATTGGGCAATGGGCGCAAGCCTGACCCAGCAACGCCGCGTGAAGGAAGAAGGCTTTCGGGTTGTAAACTTCTTTTAAGAGGGAAGAGCAGAAGACGGTACCTCTAGAATAAGCCACGGCTAACTACGTGCCAGCAGCCGCGGTAATACGTAGGTGGCAAGCGTTGTCCGGATTTACTGGGTGTAAAGGGCGTGCAGCCGGGTCTGCAAGTCAGATGTGAAATCCATGGGCTCAACCCATGAACTGCATTTGAAACTGTAGATCTTGAGTGTCGGAGGGGCAATCGGAATTCCTAGTGTAGCGGTGAAATGCGTAGATATTAGGAGGAACACCAGTGGCGAAGGCGGATTGCTGGACGATAACTGACGGTGAGGCGCGAAAGTGTGGGGAGCAAACA

>SAM.1_1 M01626:391:000000000-BWTWT:1:1104:17427:14256 1:N:0:73 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0

TGAGGAATATTGGTCAATGGGCGCTAGCCTGAACCAGCCAAGTAGCGTGAAGGATGAAGGCTCTATGGGTCGTAAACTTCTTTTATATAAGAATAAAGTGCAGTATGTATACTGTTTTGTATGTATTATATGAATAAGGATCGGCTAACTCCGTGCCAGCAGCCGCGGTAATACGGAGGATCCGAGCGTTATCCGGATTTATTGGGTTTAAAGGGAGCGTAGGTGGACTGGTAAGTCAGTTGTGAAAGTTTGCGGCTCAACCGTAAAATTGCAGTTGATACTGTCAGTCTTGAGTACAGTAGAGGTGGGCGGAATTCGTGGTGTAGCGGTGAAATGCTTAGATATCACGAAGAACTCCGATTGCGAAGGCAGCTCACTGGACTGCAACTGACACTGATGCTCGAAAGTGTGGGTATCAAACA

>SAM.1_2 M01626:391:000000000-BWTWT:1:1104:27250:14257 1:N:0:73 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0

TGAGGAATATTGGTCAATGGGCGCTAGCCTGAACCAGCCAAGTAGCGTGAAGGATGAAGGCTCTATGGGTCGTAAACTTCTTTTATATAAGAATAAAGTGCAGTATGTATACTGTTTTGTATGTATTATATGAATAAGGATCGGCTAACTCCGTGCCAGCAGCCGCGGTAATACGGAGGATCCGAGCGTTATCCGGATTTATTGGGTTTAAAGGGAGCGTAGGTGGACTGGTAAGTCAGTTGTGAAAGTTTGCGGCTCAACCGTAATATTGCAGTTGATACTGTCAGTCTTGAGTACAGTAGAGGTGGGCGGAATTCGTGGTGTAGCGGTGAAATGCTTAGATATCACGAAGAACTCCGATTGCGAAGGCAGCTCACTGGACTGCAACTGACACTGATGCTCGAAAGTGTGGGTATCAAACA

>SAM.1_3 M01626:391:000000000-BWTWT:1:1104:11048:14258 1:N:0:73 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0

TGAGGAATATTGGTCAATGGGCGCTAGCCTGAACCAGCCAAGTAGCGTGAAGGATGAAGGCTCTATGGGTCGTAAACTTCTTTTATATAAGAATAAAGTGCAGTATGTATACTGTTTTGTATGTATTATATGAATAAGGATCGGCTAACTCCGTGCCAGCAGCCGCGGTAATACGGAGGATCCGAGCGTTATCCGGATTTATTGGGTTTAAAGGGAGCGTAGGTGGACTGGTAAGTCAGTTGTGAAAGTTTGCGGCTCAACCGTAAAATTGCAGTTGATACTGGCAGTCTTGAGTACAGTAGAGGTGGGCGGAATTCGTGGTGTAGCGGTGAAATGCTTAGATATCACGAAGAACTCCGATTGCGAAGGCAGCTCACTGGACTGCAACTGACACTGATGCTCGAAAGTGTGGGTATCAAACA

>SAM.1_4 M01626:391:000000000-BWTWT:1:1104:8191:14263 1:N:0:73 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0

TGGGGAATATTGGGCAATGGAGGAAACTCTGACCCAGCAACGCCGCGTGGAGGAAGAAGGTTTTCGGATCGTAAACTCCTGTCCTTGGAGACGAGTAGAAGACGGTATCCAAGGAGGAAGCCCCGGCTAACTACGTGCCAGCAGCCGCGGTAATACGTAGGGGGCAAGCGTTGTCCAGAATAATTGGGCGTAAAGGGCGCGTAGGCGGTTTGGTAAGTCTGGAGTGAAAGCCCTGCTTTTAAGGTGGGAATTGCTTTGGATACTGTCGGGCTTGAGTGCAGTAGAGGTTAGTGGAATTCCCAGTGTAGCGGTGAAATGCGTAGAGATTGGGAGGAACACCAGTGGCGAAGGCGACTAACTGGACTGTAACTGACGCTGAGGCGCGAAAGTGTGGGGAGCAAACA

I think all files are correct.

I Have no idea why there is an error with uclust.. 

[azadeh.sa...@gmail.com]

Hello Colin

Despit of the error of uclust, I retried to run the script   

pick_de_novo_otus.py -i split_librairies_pampa2/seqs.fna -o all_de_novo_otu/uclust_picked_otus 

 and this time I had another error, but I could at least obtain a biom table, the only thing is missed is rep.tre  ( the phylogeny tree)

this time the uclust stops returning the error :

---Fatal error---

tracebackbit.cpp(64) assert failed: i > 0 && j > 0

uclust --maxrejects 32 --input /tmp/pynast_candidateSnbr05.fasta --id 0.75 --tmpdir /tmp/ --rev --maxaccepts 8 --libonly --fastapairs /tmp/uclust_alignmentsPlv5UDCg3yAUDUcxcQK9.fasta --lib /tmp/pynast_templateVLvpt9.fasta --uc /tmp/uclust_resultsdBKBgUlvGgXlUu0ZLGpV.uc

Elapsed time: 11:38

I attache you the log file,

What do you think is problem of Uclust ?

How could I create the phylogeny tree ?

[Colin Brislawn]

Hello Azadeh,

Thanks for posting that information. You have the right version of uclust, and your seqs.fna file looks OK. I'm glad you got the .biom table made successfully. 

Here is how you can build the tree, without having to repeat the OTU picking process:

http://www.wernerlab.org/teaching/qiime/overview/d 

That last ---Fatal error--- you reported happened in the align_seqs.py step of that process. When you run through all those scripts, let me know which ones work for you.

Colin

[TonyWalters]

There are literally no google hits (apart from this forum) for the error that was raised from uclust:

"tracebackbit.cpp(64) assert failed: i > 0 && j > 0"

And, as you might guess, "i" and "j" being greater than 0 isn't that informative.

Just to check a few things:

1. Are your data 16S rRNA sequences? No 18S or other genes?

2. Do you know if there are odd characters (e.g. non-ASCI) in your fasta labels?
3. Do you know if there are gaps, or odd characters in the reads themselves?

Also, you might try an alternative reference alignment, just to eliminate that (as long as your data are 16S:

http://greengenes.lbl.gov/Download/Sequence_Data/Fasta_data_files/core_set_aligned.fasta.imputed

Download this file, run:

align_seqs.py -i all_de_novo_otu/rep_set//seqs_rep_set.fasta -o all_de_novo_otu/pynast_aligned_seqs -t X

where X is the downloaded file.

[azadeh.sa...@gmail.com]

I tried to build the tree :

Here the error :

align_seqs.py -i rep_set/seqs_rep_set.fasta  -o pynast_aligned_seqs

Traceback (most recent call last):

  File "/usr/local/bin/align_seqs.py", line 211, in <module>

    main()

  File "/usr/local/bin/align_seqs.py", line 194, in main

    log_path=log_path, failure_path=failure_path)

  File "/usr/local/lib/python2.7/dist-packages/qiime/align_seqs.py", line 266, in __call__

    temp_dir=get_qiime_temp_dir())

  File "/usr/local/lib/python2.7/dist-packages/pynast/util.py", line 812, in pynast_seqs

    for seq, status in pynast_iterator:

  File "/usr/local/lib/python2.7/dist-packages/pynast/util.py", line 651, in ipynast_seqs

    current_result = pw_alignment_iterator.next()

  File "/usr/local/lib/python2.7/dist-packages/pynast/pycogent_backports/uclust.py", line 411, in uclust_search_and_align_from_fasta_filepath

    app_result = app(input_data)

  File "/usr/local/lib/python2.7/dist-packages/cogent/app/util.py", line 251, in __call__

    open(errfile).read())

cogent.app.util.ApplicationError: Unacceptable application exit status: 1

Command:

cd "/home/qiime/workspace/16S_BENJAMIN_2/all_de_novo_otu/"; uclust --maxrejects 32 --input "/tmp/pynast_candidate87q5U6.fasta" --id 0.75 --tmpdir "/tmp/" --rev --maxaccepts 8 --libonly --fastapairs "/tmp/uclust_alignmentsbOsHOaEAAkwNbmM5ZwT4.fasta" --lib "/tmp/pynast_templatebN0pjX.fasta" --uc "/tmp/uclust_resultsn6xalGiCsIY6dF51lTCx.uc" > "/tmp/tmpgtcHNPLiQYi3uZuNkLPK.txt" 2> "/tmp/tmpFgS77YoxClEr41c4ROK5.txt"

StdOut:

StdErr:

uclust v1.2.22q

(C) Copyright 2009-10 Robert C. Edgar

Licensed ONLY for use in PyNAST and QIIME.

00:00 215Mb  100.0% Reading lib, 4797 seeds

11:19 203Mb   18.2% 99.7% matched to lib at 75.0%, id 91.8%

---Fatal error---

tracebackbit.cpp(64) assert failed: i > 0 && j > 0

uclust --maxrejects 32 --input /tmp/pynast_candidate87q5U6.fasta --id 0.75 --tmpdir /tmp/ --rev --maxaccepts 8 --libonly --fastapairs /tmp/uclust_alignmentsbOsHOaEAAkwNbmM5ZwT4.fasta --lib /tmp/pynast_templatebN0pjX.fasta --uc /tmp/uclust_resultsn6xalGiCsIY6dF51lTCx.uc

Elapsed time: 11:20

 

[TonyWalters]

Yes, that's the error that was at the bottom of the previous log file too. Can you answer/try the questions/suggestion from the prior post? It might help in troubleshooting this.

[azadeh.sa...@gmail.com]

Thank you  colin and Tony for your answers,

Actually the data are 16S rRNA sequences, already filtered using 'prinseq', I got ride of all Ns and non-ASCI sequences, then they also passed filtering through the script of qiime 'Split_librairies'. I don't think there is gaps or odd things.

After downloading the file core_set_aligned.fasta.imputed and running align_seqs.py , here is the error :

align_seqs.py -i rep_set/seqs_rep_set.fasta  -o pynast_aligned_seqs -t /home/qiime/workspace/16S_BENJAMIN_2/core_set_aligned.fasta.imputed 

Traceback (most recent call last):

  File "/usr/local/bin/align_seqs.py", line 211, in <module>

    main()

  File "/usr/local/bin/align_seqs.py", line 194, in main

    log_path=log_path, failure_path=failure_path)

  File "/usr/local/lib/python2.7/dist-packages/qiime/align_seqs.py", line 266, in __call__

    temp_dir=get_qiime_temp_dir())

  File "/usr/local/lib/python2.7/dist-packages/pynast/util.py", line 812, in pynast_seqs

    for seq, status in pynast_iterator:

  File "/usr/local/lib/python2.7/dist-packages/pynast/util.py", line 651, in ipynast_seqs

    current_result = pw_alignment_iterator.next()

  File "/usr/local/lib/python2.7/dist-packages/pynast/pycogent_backports/uclust.py", line 411, in uclust_search_and_align_from_fasta_filepath

    app_result = app(input_data)

  File "/usr/local/lib/python2.7/dist-packages/cogent/app/util.py", line 251, in __call__

    open(errfile).read())

cogent.app.util.ApplicationError: Unacceptable application exit status: 1

Command:

cd "/home/qiime/workspace/16S_BENJAMIN_2/all_de_novo_otu/"; uclust --maxrejects 32 --input "/tmp/pynast_candidateWUXB_q.fasta" --id 0.75 --tmpdir "/tmp/" --rev --maxaccepts 8 --libonly --fastapairs "/tmp/uclust_alignmentsI9BtvBk0wxepIewZHzfg.fasta" --lib "/tmp/pynast_templategUcJd8.fasta" --uc "/tmp/uclust_resultsZmgBYxurFLGEQtLUK0Eg.uc" > "/tmp/tmpHGsSQow5yPYUkOpzmVHE.txt" 2> "/tmp/tmpSg2g7UyU4VgesqT4HGzD.txt"

StdOut:

StdErr:

uclust v1.2.22q

(C) Copyright 2009-10 Robert C. Edgar

Licensed ONLY for use in PyNAST and QIIME.

00:01 233Mb  100.0% Reading lib, 4938 seeds

18:42 219Mb   62.7% 99.8% matched to lib at 75.0%, id 95.1%

---Fatal error---

tracebackbit.cpp(64) assert failed: i > 0 && j > 0

uclust --maxrejects 32 --input /tmp/pynast_candidateWUXB_q.fasta --id 0.75 --tmpdir /tmp/ --rev --maxaccepts 8 --libonly --fastapairs /tmp/uclust_alignmentsI9BtvBk0wxepIewZHzfg.fasta --lib /tmp/pynast_templategUcJd8.fasta --uc /tmp/uclust_resultsZmgBYxurFLGEQtLUK0Eg.uc

Elapsed time: 18:42

   

[TonyWalters]

Okay, so that eliminates the template alignment as being at fault.

It's not giving us much of a hint as to what the source of the error is, but maybe there are details in one of the intermediate files from uclust, if it didn't delete them.

Can you see if the file:

/tmp/uclust_resultsZmgBYxurFLGEQtLUK0Eg.uc

Is still on your system? There lines at the end might indicate where in the fasta file it's running into a problem.

You might try some alternative parameters, for the sequence length/identity, e.g.:
align_seqs.py --min_length 50 --min_percent_id 0.50 -i rep_set/seqs_rep_set.fasta  -o test_parameters_pynast_aligned_seqs

and see if the same error happens.

Another possible test is to subset the data and see if the error happens for any of your fasta file or only for certain read(s). E.g., type:

head -n 2000 rep_set/seqs_rep_set.fasta > rep_set/first1000seqs.fna

align_seqs.py -i rep_set/first1000seqs.fna  -o test_pynast_aligned_seqs

and see if it errors out. If this one does error out, can you post the rep_set/first1000seqs.fna file?

[azadeh.sa...@gmail.com]

I couldn't attach the file /tmp/uclust_resultsZmgBYxurFLGEQtLUK0Eg.uc, so I copied 1000 last lines  of the file, and send it in attachement. 

I just run the test that you suggested. 

[TonyWalters]

Great, so if it is a particular sequence that is causing the error, we might get lucky and find it by looking at the sequences around the last one listed in the file, SAM.84_15454645

See if this creates a fasta file for you:

grep -A 10 -B 10 "SAM.84_15454645" rep_set/seqs_rep_set.fasta > seqs_around_SAM.84_15454645.fasta

[azadeh.sa...@gmail.com]

Hello Tony

Thank you for answer and I hope you have a nice weekend, 

I didn't find any particular characters in "SAM.84_15454645", I send you two files in attachment, 

I perform 'grep' command on rep_set ( représentative otu sets) and on seqs.fna (input sequence).

Then I see nothing abnormal ...

And still the error remains.

Thank you for your help 

Azadeh 

[TonyWalters]

Hello,

I'm traveling so I don't have much time to delve deeply into this-

can you try using the attached fasta files as input file for align_seqs.py and see if it errors out?

Did the two prior suggestions give you any different errors/messages, listed below:

[azadeh.sa...@gmail.com]

Hello Toni

I appreciate a lot your help and thank you for your time,

I tried all your suggestion :

For first thing you asked :

=====can you try using the attached fasta files as input file for align_seqs.py and see if it errors out?

I applied align_seq.py on the fasta file and it runs very well finishes without any error

========ou might try some alternative parameters, for the sequence length/identity, e.g.:

align_seqs.py --min_length 50 --min_percent_id 0.50 -i rep_set/seqs_rep_set.fasta  -o test_parameters_pynast_aligned_seqs

and see if the same error happens.

I tried also and still there is an error, align_seq stops giving an error, log file is empty, 

=========Another possible test is to subset the data and see if the error happens for any of your fasta file or only for certain read(s). E.g., type:

head -n 2000 rep_set/seqs_rep_set.fasta > rep_set/first1000seqs.fna

align_seqs.py -i rep_set/first1000seqs.fna  -o test_pynast_aligned_seqs

I tried this last suggestion, align_seq runs and finishes without any error 

-----------------------------

I'm afraid it's a problem of RAM, but it's not normal there is more than 24giga memory. ( but maybe it's not enough for whole analysis.) 

When I use the command : dmesg | grep 'Memory'

[    3.895768] Memory: 24194008K/24681016K available (7520K kernel code, 1081K rwdata, 4076K rodata, 1392K init, 1428K bss, 487008K reserved)

Thank you again,

Azadeh 

[TonyWalters]

That might be a memory issue, but it's not one of the expected error messages for memory.

Just to check the size of the file, can you run count_seqs.py (http://qiime.org/scripts/count_seqs.html) on the rep_set/seqs_rep_set.fasta file, so we can see if it's extraordinarily large?

Second, can you type the output of:

pwd

from the directory you're running these commands? I just want to check one potential issue with using the virtual box.

[azadeh.sa...@gmail.com]

Here is the number of sequence :

For  input seq.fna : 16785443

for  seqs_rep_set.fasta : 765433

pwd :

/home/qiime/workspace/16S_BENJAMIN_2

[TonyWalters]

Okay, that is an extraordinary number of reads in the _rep_set.fasta file, and probably the cause of the issue. I wouldn't expect the number of OTUs (in most cases) to exceed 10s of thousands, and certainly not be in the hundreds of thousands.

I'm not sure of the best way to trouble shoot this. Maybe try using pick_closed_reference_otus.py and see if most of the reads fail to cluster, which might indicate some artifacts in the reads (interfering with clustering and causing cluster inflation)? If you do this and get most of the reads failing, there should be a _failures.fasta file, that you can examine some reads in manually, and, for example, blast on the ncbi site to see if there are overhangs at the end(s) that are not matching the 16S rRNA gene.