Uclust failure during pick_closed_reference_otus.py and pick_open_reference_otus.py

55 views
Skip to first unread message

Ekaterina Bykova

unread,
Aug 18, 2016, 1:56:11 PM8/18/16
to Qiime 1 Forum
Hello, dear QIIME support!
I need your help with solving my problem. I searched for the issue and found some similar ones but it didn't help. I need to pick otu's for analysis in Picrust using GG v. 13.5. However, I couldn't do it, because my workflow stopped and created an error log. I also tried to performed it with default parameters (GG v.13.8 - simply excluded -r and -t from the code). 
Here are the commands I used:
echo "pick_otus:enable_rev_strand_match True"  >> $PWD/otu_picking_params_97.txt
echo "pick_otus:similarity 0.97" >> $PWD/otu_picking_params_97.txt
pick_closed_reference_otus.py -i $PWD/seqs.fna -o $PWD/ucrC97/ -p $PWD/otu_picking_params_97.txt -r $PWD/gg_13_5_otus/rep_set/97_otus.fasta -t $PWD/gg_13_5_otus/taxonomy/97_otu_taxonomy.txt
(I also tried typing the whole filepaths but it didn't help).
(And I tried to perform the same thing via simple pick_otus.py command)
(and I tried using the command on the fasta, containing seqs only from one sample)
(And I tried to validate my fasta so that it doesn't contain identical names of the seqs - I couldn't do it via validate_demultiplexed_fasta.py, because I have only seqs already passed QC without barcodes and because of it my mapping fails, but I checked the seqs names via MG-RAST and it says that they're unique)

Here is my error log.

Logging started at 05:54:31 on 18 Aug 2016
QIIME version: 1.9.1

qiime_config values:
blastmat_dir /qiime_software/blast-2.2.22-release/data
pick_otus_reference_seqs_fp /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
sc_queue all.q
pynast_template_alignment_fp /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.pynast.fasta
cluster_jobs_fp start_parallel_jobs.py
assign_taxonomy_reference_seqs_fp /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
torque_queue friendlyq
jobs_to_start 1
denoiser_min_per_core 50
assign_taxonomy_id_to_taxonomy_fp /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt
temp_dir /tmp/
blastall_fp /qiime_software/blast-2.2.22-release/bin/blastall
seconds_to_sleep 1

parameter file values:
parallel:jobs_to_start 1
pick_otus:similarity 0.97
pick_otus:otu_picking_method uclust_ref
pick_otus:enable_rev_strand_match True

Input file md5 sums:
seqs.fna: c2d6b47bdc6263c11d6bb4764b716d25
/usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta: 50b2269712b3738afb41892bed936c29
/usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt: 56ef15dccf2e931ec173f4f977ed649b

Forcing --suppress_new_clusters as this is closed-reference OTU picking.

Executing commands.

# Pick OTUs command 
pick_otus.py -i seqs.fna -o ucrC97//uclust_ref_picked_otus -r /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta -m uclust_ref --similarity 0.97 --otu_picking_method uclust_ref --enable_rev_strand_match --suppress_new_clusters



*** ERROR RAISED DURING STEP: Pick OTUs
Command run was:
 pick_otus.py -i seqs.fna -o ucrC97//uclust_ref_picked_otus -r /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta -m uclust_ref --similarity 0.97 --otu_picking_method uclust_ref --enable_rev_strand_match --suppress_new_clusters
Command returned exit status: 1
Stdout:

Stderr
Traceback (most recent call last):
  File "/usr/local/bin/pick_otus.py", line 1004, in <module>
    main()
  File "/usr/local/bin/pick_otus.py", line 924, in main
    failure_path=failure_path)
  File "/usr/local/lib/python2.7/dist-packages/qiime/pick_otus.py", line 1912, in __call__
    HALT_EXEC=HALT_EXEC)
  File "/usr/local/lib/python2.7/dist-packages/bfillings/uclust.py", line 585, in get_clusters_from_fasta_filepath
    raise ApplicationError('Error running uclust. Possible causes are '
burrito.util.ApplicationError: Error running uclust. Possible causes are unsupported version (current supported version is v1.2.22) is installed or improperly formatted input file was provided


Logging stopped at 06:25:10 on 18 Aug 2016

I checked my QIIME configs and the version of uclust by uclust --version command and it reports that I have uclust v1.2.22q (which is ok). My QIIME configs are attached. Also, I attached the test file where I copied some seqs from my 454 file. I checked this file too and, despite that it is small, it returns me the same result. Previously, I picked otu's de novo and performed the analysis (alpha- beta- diversity, assigning taxonomy and etc.) and used uclust and everything was fine.
Maybe the problem is in the amount of memory? Because I have only the laptop and cannot provide more then 1 Gb RAM or my notebook is crushing.

I decided to try picking otus via open reference and then filter, but there was no success too. I gained an error:

Logging started at 09:47:40 on 18 Aug 2016
QIIME version: 1.9.1

qiime_config values:
blastmat_dir /qiime_software/blast-2.2.22-release/data
pick_otus_reference_seqs_fp /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
sc_queue all.q
pynast_template_alignment_fp /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.pynast.fasta
cluster_jobs_fp start_parallel_jobs.py
assign_taxonomy_reference_seqs_fp /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
torque_queue friendlyq
jobs_to_start 1
denoiser_min_per_core 50
assign_taxonomy_id_to_taxonomy_fp /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt
temp_dir /tmp/
blastall_fp /qiime_software/blast-2.2.22-release/bin/blastall
seconds_to_sleep 1

parameter file values:
parallel:jobs_to_start 1

Input file md5 sums:
seqs.fna: c2d6b47bdc6263c11d6bb4764b716d25
/usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta: 50b2269712b3738afb41892bed936c29

Forcing --suppress_new_clusters as this is reference-based OTU picking.

Executing commands.

# Pick Reference OTUs command 
pick_otus.py -i seqs.fna -o pick_otus//step1_otus -r /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta -m uclust_ref  --suppress_new_clusters

Stdout:

Stderr:

# Generate full failures fasta file command 
filter_fasta.py -f seqs.fna -s pick_otus//step1_otus/seqs_failures.txt -o pick_otus//step1_otus/failures.fasta

Stdout:

Stderr:

Executing commands.

# Pick rep set command 
pick_rep_set.py -i pick_otus//step1_otus/seqs_otus.txt -o pick_otus//step1_otus/step1_rep_set.fna -f seqs.fna

Stdout:

Stderr:

Executing commands.

# Pick de novo OTUs on step1 failures command 
pick_otus.py -i pick_otus//step1_otus/failures.fasta -o pick_otus//step4_otus/ -m uclust  --denovo_otu_id_prefix New.CleanUp.ReferenceOTU

Stdout:

Stderr:

# Merge OTU maps command 
cat pick_otus//step1_otus/seqs_otus.txt  pick_otus//step4_otus//failures_otus.txt > pick_otus//final_otu_map.txt

Stdout:

Stderr:

# Pick representative set for subsampled failures command 
pick_rep_set.py -i pick_otus//step4_otus//failures_otus.txt -o pick_otus//step4_otus//step4_rep_set.fna -f pick_otus//step1_otus/failures.fasta

Stdout:

Stderr:

# Filter singletons from the otu map using API 
python -c "import qiime; qiime.filter.filter_otus_from_otu_map('/home/qiime/Desktop/Picrust/pick_otus/final_otu_map.txt', '/home/qiime/Desktop/Picrust/pick_otus/final_otu_map_mc2.txt', '2')"

# Write non-singleton otus representative sequences from step1 to the final rep set file: pick_otus//rep_set.fna

# Copy the full input refseqs file to the new refseq file
cp /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta pick_otus//new_refseqs.fna

# Write non-singleton otus representative sequences from step 4 to the final representative set and the new reference set (pick_otus//rep_set.fna and pick_otus//new_refseqs.fna respectively)

Executing commands.

# Make the otu table command 
make_otu_table.py -i pick_otus//final_otu_map_mc2.txt -o pick_otus//otu_table_mc2.biom

Stdout:

Stderr:

Executing commands.

# Assign taxonomy command 
assign_taxonomy.py -o pick_otus//uclust_assigned_taxonomy -i pick_otus//rep_set.fna 



*** ERROR RAISED DURING STEP: Assign taxonomy
Command run was:
 assign_taxonomy.py -o pick_otus//uclust_assigned_taxonomy -i pick_otus//rep_set.fna 
Command returned exit status: 1
Stdout:

Stderr
Traceback (most recent call last):
  File "/usr/local/bin/assign_taxonomy.py", line 417, in <module>
    main()
  File "/usr/local/bin/assign_taxonomy.py", line 394, in main
    log_path=log_path)
  File "/usr/local/lib/python2.7/dist-packages/qiime/assign_taxonomy.py", line 1304, in __call__
    '--uc': uc_path})
  File "/usr/local/lib/python2.7/dist-packages/burrito/util.py", line 285, in __call__
    'StdErr:\n%s\n' % open(errfile).read())
burrito.util.ApplicationError: Unacceptable application exit status: 137
Command:
cd "/home/qiime/Desktop/Picrust/"; uclust --input "pick_otus//rep_set.fna" --id 0.9 --rev --maxaccepts 3 --allhits --libonly --lib "/usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta" --uc "/tmp/UclustConsensusTaxonAssigner_zLHm0d.uc" > "/tmp/tmpEK9MzFNhaRkIojtLipeI.txt" 2> "/tmp/tmpZwRdFQXsTCJUfPrIuBtE.txt"
StdOut:

StdErr:
uclust v1.2.22q
(C) Copyright 2009-10 Robert C. Edgar
Licensed ONLY for use in PyNAST and QIIME.
00:00  37Mb    0.0% Reading lib, 0 seeds
00:01 199Mb    4.4% Reading lib, 4391 seeds
00:02 351Mb    8.4% Reading lib, 8372 seeds
00:03 486Mb   12.0% Reading lib, 11923 seeds
00:04 619Mb   15.6% Reading lib, 15502 seeds
00:05 762Mb   19.2% Reading lib, 19193 seeds
00:06 877Mb   22.4% Reading lib, 22358 seeds
00:07 930Mb   23.9% Reading lib, 23817 seeds
00:08 1.0Gb   25.9% Reading lib, 25840 seeds
00:09 1.1Gb   27.1% Reading lib, 27102 seeds
00:10 1.1Gb   27.3% Reading lib, 27220 seeds
00:11 1.1Gb   28.2% Reading lib, 28228 seeds
00:12 1.1Gb   28.4% Reading lib, 28385 seeds
00:13 1.2Gb   30.9% Reading lib, 30899 seeds
00:14 1.3Gb   34.5% Reading lib, 34492 seeds
00:16 1.4Gb   35.2% Reading lib, 35210 seeds
00:17 1.4Gb   36.5% Reading lib, 36508 seeds
00:18 1.5Gb   38.5% Reading lib, 38485 seeds
00:19 1.6Gb   40.5% Reading lib, 40419 seeds
00:20 1.6Gb   41.6% Reading lib, 41606 seeds
00:21 1.6Gb   42.2% Reading lib, 42167 seeds
00:22 1.6Gb   42.4% Reading lib, 42415 seeds
00:23 1.7Gb   42.8% Reading lib, 42833 seeds
00:25 1.7Gb   43.1% Reading lib, 43107 seeds
00:26 1.7Gb   43.6% Reading lib, 43616 seeds
00:27 1.7Gb   44.2% Reading lib, 44216 seeds
00:28 1.8Gb   45.3% Reading lib, 45308 seeds
00:29 1.8Gb   46.6% Reading lib, 46600 seeds
00:33 1.8Gb   46.9% Reading lib, 46886 seeds
00:34 1.9Gb   48.1% Reading lib, 48085 seeds
00:35 1.9Gb   48.7% Reading lib, 48669 seeds
00:36 1.9Gb   49.4% Reading lib, 49416 seeds
00:37 1.9Gb   49.6% Reading lib, 49568 seeds
00:38 1.9Gb   50.3% Reading lib, 50244 seeds
00:39 2.0Gb   50.9% Reading lib, 50848 seeds
00:40 2.0Gb   51.2% Reading lib, 51136 seeds
00:42 2.0Gb   51.6% Reading lib, 51524 seeds
00:42 2.0Gb   51.6% Reading lib, 51574 seeds
00:44 2.0Gb   52.6% Reading lib, 52563 seeds
00:45 2.0Gb   52.9% Reading lib, 52852 seeds
00:46 2.0Gb   53.2% Reading lib, 53093 seeds
00:47 2.1Gb   53.5% Reading lib, 53443 seeds
00:48 2.1Gb   53.9% Reading lib, 53770 seeds
00:49 2.1Gb   54.7% Reading lib, 54538 seeds
00:50 2.1Gb   54.9% Reading lib, 54776 seeds
00:51 2.1Gb   55.0% Reading lib, 54864 seeds
00:52 2.1Gb   55.0% Reading lib, 54914 seeds
00:53 2.1Gb   55.1% Reading lib, 54952 seeds
00:54 2.1Gb   55.3% Reading lib, 55189 seeds
00:55 2.2Gb   55.8% Reading lib, 55702 seeds
00:56 2.2Gb   56.4% Reading lib, 56205 seeds
00:57 2.2Gb   56.5% Reading lib, 56307 seeds
01:00 2.2Gb   57.1% Reading lib, 56917 seeds
01:03 2.2Gb   57.2% Reading lib, 57003 seeds
01:04 2.2Gb   57.2% Reading lib, 57030 seeds
01:06 2.2Gb   57.7% Reading lib, 57535 seeds
01:06 2.2Gb   57.9% Reading lib, 57712 seeds
01:07 2.2Gb   58.1% Reading lib, 57889 seeds
01:09 2.3Gb   58.4% Reading lib, 58221 seeds
01:17 2.3Gb   59.0% Reading lib, 58840 seeds
01:18 2.3Gb   59.1% Reading lib, 58962 seeds
01:21 2.3Gb   59.2% Reading lib, 59051 seeds
01:24 2.3Gb   59.6% Reading lib, 59454 seeds
01:26 2.3Gb   60.2% Reading lib, 59975 seeds
01:27 2.3Gb   60.2% Reading lib, 60023 seeds
01:32 2.3Gb   60.4% Reading lib, 60170 seeds
01:35 2.3Gb   60.7% Reading lib, 60468 seeds
01:40 2.4Gb   61.0% Reading lib, 60744 seeds
01:41 2.4Gb   61.1% Reading lib, 60936 seeds
01:44 2.4Gb   61.4% Reading lib, 61226 seeds
01:49 2.4Gb   61.9% Reading lib, 61639 seeds
01:49 2.4Gb   61.9% Reading lib, 61654 seeds
01:50 2.4Gb   62.0% Reading lib, 61739 seeds
01:52 2.4Gb   62.0% Reading lib, 61797 seeds
01:54 2.4Gb   62.1% Reading lib, 61919 seeds
01:55 2.4Gb   62.2% Reading lib, 61981 seeds
01:56 2.4Gb   62.3% Reading lib, 62088 seeds
02:12 2.4Gb   62.8% Reading lib, 62586 seeds
02:20 2.4Gb   62.9% Reading lib, 62693 seeds
02:21 2.4Gb   63.0% Reading lib, 62727 seeds
02:22 2.4Gb   63.0% Reading lib, 62757 seeds
02:23 2.4Gb   63.0% Reading lib, 62798 seeds
02:24 2.4Gb   63.1% Reading lib, 62877 seeds
02:28 2.4Gb   63.2% Reading lib, 62971 seeds
02:29 2.4Gb   63.3% Reading lib, 63031 seeds
02:30 2.4Gb   63.3% Reading lib, 63091 seeds
02:37 2.5Gb   63.7% Reading lib, 63438 seeds
02:38 2.5Gb   63.8% Reading lib, 63535 seeds
02:40 2.5Gb   63.8% Reading lib, 63577 seeds
02:41 2.5Gb   63.9% Reading lib, 63620 seeds
02:42 2.5Gb   64.0% Reading lib, 63727 seeds
02:45 2.5Gb   64.0% Reading lib, 63777 seeds
02:46 2.5Gb   64.0% Reading lib, 63784 seeds
02:47 2.5Gb   64.0% Reading lib, 63791 seeds
02:47 2.5Gb   64.1% Reading lib, 63806 seeds
02:51 2.5Gb   64.1% Reading lib, 63858 seeds
02:52 2.5Gb   64.1% Reading lib, 63890 seeds
02:54 2.5Gb   64.2% Reading lib, 63975 seeds
02:55 2.5Gb   64.2% Reading lib, 63988 seeds
02:56 2.5Gb   64.4% Reading lib, 64120 seeds
02:59 2.5Gb   64.7% Reading lib, 64449 seeds
03:00 2.5Gb   64.8% Reading lib, 64509 seeds
03:01 2.5Gb   64.9% Reading lib, 64609 seeds
03:05 2.5Gb   65.2% Reading lib, 64925 seeds
03:07 2.5Gb   65.4% Reading lib, 65123 seeds
03:08 2.5Gb   65.5% Reading lib, 65189 seeds
03:17 2.5Gb   66.0% Reading lib, 65705 seeds
03:18 2.5Gb   66.1% Reading lib, 65783 seeds
03:23 2.5Gb   66.2% Reading lib, 65882 seeds
03:25 2.6Gb   66.2% Reading lib, 65940 seeds
03:32 2.6Gb   66.6% Reading lib, 66363 seeds
03:33 2.6Gb   66.7% Reading lib, 66397 seeds
03:34 2.6Gb   66.7% Reading lib, 66417 seeds
03:36 2.6Gb   66.8% Reading lib, 66525 seeds
03:37 2.6Gb   66.9% Reading lib, 66597 seeds
03:38 2.6Gb   66.9% Reading lib, 66663 seeds
03:39 2.6Gb   67.1% Reading lib, 66865 seeds
03:50 2.6Gb   67.7% Reading lib, 67385 seeds
03:51 2.6Gb   67.8% Reading lib, 67510 seeds
03:52 2.6Gb   68.0% Reading lib, 67724 seeds
03:53 2.6Gb   68.1% Reading lib, 67771 seeds
03:56 2.6Gb   68.2% Reading lib, 67913 seeds
03:57 2.6Gb   68.3% Reading lib, 67970 seeds
03:58 2.6Gb   68.4% Reading lib, 68078 seeds
03:59 2.6Gb   68.4% Reading lib, 68134 seeds
04:00 2.6Gb   68.7% Reading lib, 68351 seeds
04:04 2.7Gb   68.9% Reading lib, 68567 seeds
04:06 2.7Gb   69.1% Reading lib, 68803 seeds
04:07 2.7Gb   69.2% Reading lib, 68849 seeds
04:09 2.7Gb   69.4% Reading lib, 69091 seeds
04:12 2.7Gb   69.5% Reading lib, 69153 seeds
04:13 2.7Gb   69.5% Reading lib, 69221 seeds
04:15 2.7Gb   69.7% Reading lib, 69416 seeds
04:16 2.7Gb   69.8% Reading lib, 69450 seeds
04:17 2.7Gb   69.9% Reading lib, 69592 seeds
04:25 2.7Gb   70.2% Reading lib, 69888 seeds
04:27 2.7Gb   70.2% Reading lib, 69907 seeds
04:28 2.7Gb   70.2% Reading lib, 69929 seeds
04:30 2.7Gb   70.3% Reading lib, 70019 seeds
04:31 2.7Gb   70.4% Reading lib, 70104 seeds
04:32 2.7Gb   70.4% Reading lib, 70126 seeds
04:34 2.7Gb   70.5% Reading lib, 70202 seeds
04:35 2.7Gb   70.7% Reading lib, 70372 seeds
04:37 2.7Gb   70.9% Reading lib, 70576 seeds
04:40 2.7Gb   71.1% Reading lib, 70740 seeds
04:41 2.7Gb   71.1% Reading lib, 70813 seeds
04:42 2.7Gb   71.3% Reading lib, 70929 seeds
04:43 2.7Gb   71.3% Reading lib, 70987 seeds
04:50 2.8Gb   71.7% Reading lib, 71384 seeds
04:51 2.8Gb   71.7% Reading lib, 71410 seeds
04:52 2.8Gb   71.8% Reading lib, 71487 seeds
04:54 2.8Gb   71.9% Reading lib, 71539 seeds
04:55 2.8Gb   72.0% Reading lib, 71643 seeds
05:12 2.8Gb   72.7% Reading lib, 72306 seeds
05:13 2.8Gb   72.7% Reading lib, 72312 seeds
05:15 2.8Gb   72.7% Reading lib, 72340 seeds
05:16 2.8Gb   72.8% Reading lib, 72416 seeds
05:18 2.8Gb   72.8% Reading lib, 72454 seeds
05:19 2.8Gb   72.8% Reading lib, 72458 seeds
05:25 2.8Gb   72.9% Reading lib, 72545 seeds
05:26 2.8Gb   72.9% Reading lib, 72549 seeds
05:27 2.8Gb   72.9% Reading lib, 72593 seeds
05:29 2.8Gb   73.0% Reading lib, 72606 seeds
05:30 2.8Gb   73.0% Reading lib, 72610 seeds
05:31 2.8Gb   73.0% Reading lib, 72644 seeds
05:32 2.8Gb   73.1% Reading lib, 72716 seeds
05:41 2.8Gb   73.5% Reading lib, 73127 seeds
05:50 2.8Gb   73.8% Reading lib, 73414 seeds
05:51 2.8Gb   73.8% Reading lib, 73432 seeds
05:52 2.8Gb   73.8% Reading lib, 73446 seeds
05:54 2.8Gb   73.9% Reading lib, 73496 seeds
05:56 2.8Gb   73.9% Reading lib, 73509 seeds
05:58 2.8Gb   73.9% Reading lib, 73519 seeds
05:59 2.8Gb   73.9% Reading lib, 73553 seeds
06:00 2.8Gb   73.9% Reading lib, 73587 seeds
06:02 2.8Gb   74.0% Reading lib, 73628 seeds
06:03 2.8Gb   74.0% Reading lib, 73661 seeds
06:04 2.8Gb   74.1% Reading lib, 73697 seeds
06:07 2.8Gb   74.2% Reading lib, 73817 seeds
06:08 2.8Gb   74.3% Reading lib, 73884 seeds
06:09 2.9Gb   74.3% Reading lib, 73930 seeds
06:24 2.9Gb   74.8% Reading lib, 74478 seeds
06:24 2.9Gb   74.8% Reading lib, 74479 seeds
06:27 2.9Gb   74.9% Reading lib, 74525 seeds
06:30 2.9Gb   74.9% Reading lib, 74557 seeds
06:31 2.9Gb   74.9% Reading lib, 74570 seeds
06:32 2.9Gb   75.0% Reading lib, 74592 seeds
06:34 2.9Gb   75.0% Reading lib, 74639 seeds
06:35 2.9Gb   75.0% Reading lib, 74653 seeds
06:36 2.9Gb   75.1% Reading lib, 74696 seeds
06:37 2.9Gb   75.1% Reading lib, 74699 seeds
06:55 2.9Gb   75.4% Reading lib, 75016 seeds
06:58 2.9Gb   75.4% Reading lib, 75033 seeds
06:59 2.9Gb   75.4% Reading lib, 75054 seeds
07:01 2.9Gb   75.5% Reading lib, 75078 seeds
07:04 2.9Gb   75.5% Reading lib, 75097 seeds
07:04 2.9Gb   75.5% Reading lib, 75100 seeds
07:05 2.9Gb   75.5% Reading lib, 75110 seeds
07:06 2.9Gb   75.5% Reading lib, 75116 seeds
07:07 2.9Gb   75.5% Reading lib, 75123 seeds
07:08 2.9Gb   75.5% Reading lib, 75127 seeds
07:09 2.9Gb   75.5% Reading lib, 75140 seeds
07:11 2.9Gb   75.5% Reading lib, 75165 seeds
07:12 2.9Gb   75.6% Reading lib, 75180 seeds
07:14 2.9Gb   75.6% Reading lib, 75203 seeds
07:15 2.9Gb   75.6% Reading lib, 75214 seeds
07:18 2.9Gb   75.6% Reading lib, 75266 seeds
07:20 2.9Gb   75.7% Reading lib, 75283 seeds
07:21 2.9Gb   75.7% Reading lib, 75303 seeds
07:22 2.9Gb   75.7% Reading lib, 75336 seeds
07:23 2.9Gb   75.7% Reading lib, 75340 seeds
07:26 2.9Gb   75.8% Reading lib, 75417 seeds
07:27 2.9Gb   75.8% Reading lib, 75442 seeds
07:28 2.9Gb   75.8% Reading lib, 75469 seeds
07:31 2.9Gb   75.9% Reading lib, 75554 seeds
07:35 2.9Gb   76.0% Reading lib, 75608 seeds
07:40 2.9Gb   76.0% Reading lib, 75662 seeds
07:41 2.9Gb   76.1% Reading lib, 75672 seeds
07:42 2.9Gb   76.1% Reading lib, 75697 seeds
07:43 2.9Gb   76.1% Reading lib, 75708 seeds
07:44 2.9Gb   76.1% Reading lib, 75716 seeds
07:45 2.9Gb   76.1% Reading lib, 75732 seeds
07:46 2.9Gb   76.1% Reading lib, 75740 seeds
07:48 2.9Gb   76.2% Reading lib, 75765 seeds
07:48 2.9Gb   76.2% Reading lib, 75768 seeds
07:49 2.9Gb   76.2% Reading lib, 75782 seeds
07:51 2.9Gb   76.2% Reading lib, 75825 seeds
07:57 2.9Gb   76.3% Reading lib, 75945 seeds
08:00 2.9Gb   76.3% Reading lib, 75954 seeds
08:01 2.9Gb   76.3% Reading lib, 75955 seeds
08:02 2.9Gb   76.4% Reading lib, 75972 seeds
08:03 2.9Gb   76.4% Reading lib, 75983 seeds
08:05 2.9Gb   76.4% Reading lib, 75994 seeds
08:06 2.9Gb   76.4% Reading lib, 75999 seeds
08:11 2.9Gb   76.4% Reading lib, 76021 seeds
08:12 2.9Gb   76.4% Reading lib, 76025 seeds
08:13 2.9Gb   76.4% Reading lib, 76045 seeds
08:15 2.9Gb   76.5% Reading lib, 76064 seeds
08:16 2.9Gb   76.5% Reading lib, 76102 seeds
08:18 2.9Gb   76.6% Reading lib, 76192 seeds
08:19 2.9Gb   76.6% Reading lib, 76224 seeds
08:20 2.9Gb   76.7% Reading lib, 76262 seeds
08:22 2.9Gb   76.7% Reading lib, 76300 seeds
08:23 2.9Gb   76.8% Reading lib, 76363 seeds
08:25 2.9Gb   76.8% Reading lib, 76385 seeds
08:26 2.9Gb   76.8% Reading lib, 76442 seeds
08:28 2.9Gb   77.0% Reading lib, 76584 seeds
08:30 2.9Gb   77.1% Reading lib, 76678 seeds
08:32 2.9Gb   77.1% Reading lib, 76748 seeds
08:33 2.9Gb   77.2% Reading lib, 76764 seeds
08:36 2.9Gb   77.2% Reading lib, 76789 seeds
08:41 2.9Gb   77.2% Reading lib, 76846 seeds
08:42 3.0Gb   77.3% Reading lib, 76854 seeds
08:43 3.0Gb   77.3% Reading lib, 76873 seeds
08:47 3.0Gb   77.4% Reading lib, 76951 seeds
08:49 3.0Gb   77.4% Reading lib, 76955 seeds
08:53 3.0Gb   77.4% Reading lib, 76957 seeds
08:53 3.0Gb   77.4% Reading lib, 76958 seeds
08:55 3.0Gb   77.4% Reading lib, 76960 seeds
08:56 3.0Gb   77.4% Reading lib, 76961 seeds
08:56 3.0Gb   77.4% Reading lib, 76962 seeds
08:58 3.0Gb   77.4% Reading lib, 76966 seeds
09:00 3.0Gb   77.4% Reading lib, 76971 seeds
09:02 3.0Gb   77.4% Reading lib, 76978 seeds
09:02 3.0Gb   77.4% Reading lib, 76980 seeds
09:03 3.0Gb   77.4% Reading lib, 76986 seeds
09:04 3.0Gb   77.4% Reading lib, 76994 seeds
09:06 3.0Gb   77.4% Reading lib, 77018 seeds
09:09 3.0Gb   77.5% Reading lib, 77068 seeds
09:11 3.0Gb   77.5% Reading lib, 77147 seeds
09:13 3.0Gb   77.6% Reading lib, 77165 seeds
09:13 3.0Gb   77.6% Reading lib, 77176 seeds
09:14 3.0Gb   77.6% Reading lib, 77190 seeds
09:21 3.0Gb   77.7% Reading lib, 77273 seeds
09:21 3.0Gb   77.7% Reading lib, 77277 seeds
09:22 3.0Gb   77.7% Reading lib, 77279 seeds
09:23 3.0Gb   77.7% Reading lib, 77297 seeds
09:24 3.0Gb   77.7% Reading lib, 77316 seeds
09:25 3.0Gb   77.7% Reading lib, 77324 seeds
09:32 3.0Gb   77.8% Reading lib, 77402 seeds
09:39 3.0Gb   77.8% Reading lib, 77440 seeds
09:40 3.0Gb   77.8% Reading lib, 77441 seeds
09:41 3.0Gb   77.8% Reading lib, 77442 seeds
09:44 3.0Gb   77.9% Reading lib, 77449 seeds
09:50 3.0Gb   77.9% Reading lib, 77451 seeds
09:53 3.0Gb   77.9% Reading lib, 77452 seeds
09:54 3.0Gb   77.9% Reading lib, 77453 seeds
09:55 3.0Gb   77.9% Reading lib, 77454 seeds
09:56 3.0Gb   77.9% Reading lib, 77455 seeds
10:05 3.0Gb   77.9% Reading lib, 77457 seeds
10:13 3.0Gb   77.9% Reading lib, 77458 seeds
10:23 3.0Gb   77.9% Reading lib, 77459 seeds
10:27 3.0Gb   77.9% Reading lib, 77460 seeds
10:31 3.0Gb   77.9% Reading lib, 77461 seeds
10:34 3.0Gb   77.9% Reading lib, 77462 seeds
10:34 3.0Gb   77.9% Reading lib, 77463 seeds
10:36 3.0Gb   77.9% Reading lib, 77465 seeds
10:46 3.0Gb   77.9% Reading lib, 77472 seeds
10:49 3.0Gb   77.9% Reading lib, 77473 seeds
10:50 3.0Gb   77.9% Reading lib, 77474 seeds
10:52 3.0Gb   77.9% Reading lib, 77487 seeds
10:53 3.0Gb   77.9% Reading lib, 77495 seeds
10:54 3.0Gb   77.9% Reading lib, 77503 seeds
10:55 3.0Gb   77.9% Reading lib, 77507 seeds
10:57 3.0Gb   77.9% Reading lib, 77522 seeds
10:58 3.0Gb   77.9% Reading lib, 77526 seeds
10:59 3.0Gb   77.9% Reading lib, 77543 seeds
11:00 3.0Gb   78.0% Reading lib, 77557 seeds
11:04 3.0Gb   78.0% Reading lib, 77572 seeds
11:05 3.0Gb   78.0% Reading lib, 77579 seeds
11:07 3.0Gb   78.0% Reading lib, 77605 seeds
11:08 3.0Gb   78.0% Reading lib, 77616 seeds
11:21 3.0Gb   78.1% Reading lib, 77684 seeds
11:21 3.0Gb   78.1% Reading lib, 77685 seeds
11:22 3.0Gb   78.1% Reading lib, 77686 seeds
11:24 3.0Gb   78.1% Reading lib, 77689 seeds
11:28 3.0Gb   78.1% Reading lib, 77694 seeds
11:28 3.0Gb   78.1% Reading lib, 77695 seeds
11:29 3.0Gb   78.1% Reading lib, 77696 seeds
11:32 3.0Gb   78.1% Reading lib, 77697 seeds
11:34 3.0Gb   78.1% Reading lib, 77698 seeds
11:35 3.0Gb   78.1% Reading lib, 77700 seeds
11:37 3.0Gb   78.1% Reading lib, 77701 seeds
11:43 3.0Gb   78.1% Reading lib, 77704 seeds
11:45 3.0Gb   78.1% Reading lib, 77705 seeds
11:46 3.0Gb   78.1% Reading lib, 77706 seeds
11:48 3.0Gb   78.1% Reading lib, 77710 seeds
11:48 3.0Gb   78.1% Reading lib, 77711 seeds
11:50 3.0Gb   78.1% Reading lib, 77732 seeds
11:51 3.0Gb   78.1% Reading lib, 77734 seeds
11:55 3.0Gb   78.2% Reading lib, 77759 seeds
11:55 3.0Gb   78.2% Reading lib, 77761 seeds
11:57 3.0Gb   78.2% Reading lib, 77769 seeds
11:58 3.0Gb   78.2% Reading lib, 77776 seeds
12:01 3.0Gb   78.2% Reading lib, 77788 seeds
12:01 3.0Gb   78.2% Reading lib, 77789 seeds
12:02 3.0Gb   78.2% Reading lib, 77790 seeds
12:04 3.0Gb   78.2% Reading lib, 77810 seeds
12:08 3.0Gb   78.2% Reading lib, 77819 seeds
12:08 3.0Gb   78.2% Reading lib, 77820 seeds
12:13 3.0Gb   78.2% Reading lib, 77835 seeds
12:16 3.0Gb   78.2% Reading lib, 77836 seeds
12:16 3.0Gb   78.2% Reading lib, 77837 seeds
12:17 3.0Gb   78.3% Reading lib, 77845 seeds
12:19 3.0Gb   78.3% Reading lib, 77878 seeds
12:20 3.0Gb   78.3% Reading lib, 77886 seeds
12:22 3.0Gb   78.3% Reading lib, 77908 seeds
12:23 3.0Gb   78.3% Reading lib, 77915 seeds
12:24 3.0Gb   78.3% Reading lib, 77926 seeds
12:25 3.0Gb   78.3% Reading lib, 77933 seeds
12:32 3.0Gb   78.4% Reading lib, 77957 seeds
12:32 3.0Gb   78.4% Reading lib, 77958 seeds
12:35 3.0Gb   78.4% Reading lib, 77964 seeds
12:39 3.0Gb   78.4% Reading lib, 77965 seeds
12:40 3.0Gb   78.4% Reading lib, 77966 seeds
12:42 3.0Gb   78.4% Reading lib, 77967 seeds
12:47 3.0Gb   78.4% Reading lib, 77968 seeds
12:48 3.0Gb   78.4% Reading lib, 77969 seeds
12:52 3.0Gb   78.4% Reading lib, 77970 seeds
13:00 3.0Gb   78.4% Reading lib, 77971 seeds
13:05 3.0Gb   78.4% Reading lib, 77972 seeds
13:10 3.0Gb   78.4% Reading lib, 77973 seeds
13:18 3.0Gb   78.4% Reading lib, 77974 seeds
13:27 3.0Gb   78.4% Reading lib, 77975 seeds
13:36 3.0Gb   78.4% Reading lib, 77976 seeds
13:42 3.0Gb   78.4% Reading lib, 77977 seeds
13:49 3.0Gb   78.4% Reading lib, 77978 seeds
14:01 3.0Gb   78.4% Reading lib, 77979 seeds
14:20 3.0Gb   78.4% Reading lib, 77980 seeds
14:40 3.0Gb   78.4% Reading lib, 77981 seeds
Killed

My seqs.fna is 26 Mb, which, I suppose, is rather small.
Thank you for your help.
QIIME conf
test_seq.fna

Colin Brislawn

unread,
Aug 18, 2016, 3:35:01 PM8/18/16
to qiime...@googlegroups.com
Good afternoon!

Thank you for the super detailed log file and all these notes attached. I read through your log file and tried to find that last script that it tried to run:
 assign_taxonomy.py -o pick_otus//uclust_assigned_taxonomy -i pick_otus//rep_set.fna 

OK, so it got all the way to assign_taxonomy.py, then failed with this message:
14:20 3.0Gb   78.4% Reading lib, 77980 seeds
14:40 3.0Gb   78.4% Reading lib, 77981 seeds
Killed
Wow, so it took 20 seconds to read in the final read (77981) before it crashed! Maybe this script was running out of RAM and using the swap disk instead...

Given that the database read in 3.0 GB and your computer only has 1 GB, this is almost certainly the problem. Do you have a larger computer on which to run this analysis? 

I hope this helps,
Colin



Ekaterina Bykova

unread,
Aug 18, 2016, 4:03:38 PM8/18/16
to Qiime 1 Forum
Thank you for quick answering! Unfortunately, I have only this computer now =(
Okay, I understand why I failed with opened reference. But why the closed reference otu picking (see the log above) failed too, even on a small test_seq.fna? It failes on the uclust_ref stage. Is it also because of RAM?

Ekaterina Bykova

unread,
Aug 19, 2016, 4:35:55 AM8/19/16
to qiime...@googlegroups.com
Hi, Colin
The night passed and i desided to try pick_closed_reference_otus.py again. It must be some sort of magic (or maybe my curses), but today it worked (somehow, I don't know how). BIOM summary looks normal.
Num samples: 8
Num observations: 2096
Total count: 25204
Table density (fraction of non-zero values): 0.205

Counts/sample summary:
 Min: 1229.0
 Max: 5148.0
 Median: 3169.000
 Mean: 3150.500
 Std. dev.: 1380.996
 Sample Metadata Categories: None provided
 Observation Metadata Categories: taxonomy

Counts/sample detail:
 merz: 1229.0
 merzchit: 1287.0
 sovr: 2382.0
 cast3500: 2913.0
 cast4500: 3425.0
 sovrchit: 4088.0
 cast4500chit: 4732.0
 cast3500chit: 5148.0

BUT unfortunately I need your help again. Looks like Picrust still works with BIOM format 1.3.1 and mine is 2.1.4. I tried to reinstall BIOM, with 
pip uninstall biom-format
and then wanted to run
But it looks like I just can't simply uninstall it and install the older version (it gives me loads of angry red text). Is there any way to convert my BIOM to 1.3.1 in QIIME?

(Picrust forum has many issues like mine but no answers how to manage with it, except pip uninstall)

Ekaterina Bykova

unread,
Aug 19, 2016, 4:50:51 AM8/19/16
to Qiime 1 Forum
Hi, all
Looks like the problem is solved
I just needed to convert biom from hdf5 to json and Picrust accepted it (no need to change the version of your BIOM in QIIME)
Just type 
biom convert -i [your table here] --to-json -o my_table_json.biom

Colin Brislawn

unread,
Aug 19, 2016, 1:27:52 PM8/19/16
to Qiime 1 Forum
Oh great! I'm glad that OTU picking started working again, and you were able to format your table for PICRUSt.

Thanks for posting how you resolved this issue by converting your biom table to JSON format. This will help future users on the forum.

Colin

Reply all
Reply to author
Forward
0 new messages