Problem with ampliconnoise.py and denoise_wrapper.py

[Martin]

Hi,

I've been trying to denoise my data with either ampliconnoise or
denoise_wrapper, but unfortunately neither of them works, although
split_libraries.py seems to work fine. I checked the mapping file and
went through the earlier forum entries that reported the same errors,
but none of the suggestions seems to work for me, and I can't find any
mistakes in the mapping file.

The ampliconnoise log file contains the following:

Logging started at 08:35:10 on 06 Sep 2011
QIIME version: 1.3.0

qiime_config values:
blastmat_dir /software/blast-2.2.22-release/data
pynast_template_alignment_fp /software/core_set_aligned.fasta.imputed
cluster_jobs_fp /software/qiime-1.2.0-release/bin/
start_parallel_jobs.py
torque_queue friendlyq
template_alignment_lanemask_fp /software/lanemask_in_1s_and_0s
jobs_to_start 1
cloud_environment True
qiime_scripts_dir /software/qiime-1.2.0-release/bin
denoiser_min_per_core 50
working_dir /tmp/
python_exe_fp /software/python-2.6.4-release/bin/python
temp_dir /tmp/
blastall_fp /software/blast-2.2.22-release/bin/blastall
seconds_to_sleep 60

parameter file values:

Executing commands.

# change to output dir command
cd /home/qiime/Desktop/test_ampliconnoise/ampliconnoise_dir

Stdout:

Stderr:

# confirm pyro lookup filepath environment variable command
echo $PYRO_LOOKUP_FILE > pyro_lookup_filepath.txt

Stdout:

Stderr:

# split sff.txt via barcodes (keys) command
SplitKeys.pl GAGTTTGATC[ACTG]TGGCTCAG map.csv < /home/qiime/Desktop/
test_ampliconnoise/NvesL.sff.txt > splitkeys_log.txt 2> unassigned.fna

Stdout:

Stderr:

# clean flows NvesL command
Clean360.pl GAGTTTGATC[ACTG]TGGCTCAG NvesL < NvesL.raw

Stdout:

Stderr:

# pyrodist NvesL command
mpirun -np 2 PyroDist -in NvesL.dat -out NvesL > NvesL.pdout



*** ERROR RAISED DURING STEP: pyrodist NvesL
Command run was:
mpirun -np 2 PyroDist -in NvesL.dat -out NvesL > NvesL.pdout
Command returned exit status: 255
Stdout:

Stderr


Logging stopped at 08:35:21 on 06 Sep 2011
----------------------------------------------------------------

The error message for denoise_wrapper is:

Traceback (most recent call last):
File "/software/qiime-1.3.0-release/bin/denoise_wrapper.py", line
159, in <module>
main()
File "/software/qiime-1.3.0-release/bin/denoise_wrapper.py", line
145, in main
titanium=opts.titanium)
File "/software/qiime-1.3.0-release/lib/qiime/denoise_wrapper.py",
line 37, in fast_denoiser
verbose=verbose, titanium=titanium)
File "/software/qiime-1.3.0-release/lib/qiime/denoiser/
flowgram_clustering.py", line 612, in denoise_seqs
verbose=verbose, squeeze=squeeze, primer=primer)
File "/software/qiime-1.3.0-release/lib/qiime/denoiser/
preprocess.py", line 202, in preprocess
barcode_mapping = extract_barcodes_from_mapping(labels)
File "/software/qiime-1.3.0-release/lib/qiime/denoiser/
flowgram_filter.py", line 259, in extract_barcodes_from_mapping
flowgram_id = tmatch.group(2)
AttributeError: 'NoneType' object has no attribute 'group'

-----------------------------------------------------------------

My mapping file looks like this:

#SampleID BarcodeSequence LinkerPrimerSequence Description
NvesL AGTCCCAA GAGTTTGATCNTGGCTCAG NvesL
NvesP AGTCGTCA GAGTTTGATCNTGGCTCAG NvesP

------------------------------------------------------------------

I would really appreciate any suggestions to help me denoise the data.

Thanks,
Martin

[Jens Reeder]

Hi Martin

Can you post the command that you used for denoise_wrapper.
It looks like you passed in the wrong fasta file. It must be the
output fasta file from split_libraries.

Jens

[Martin]

Hi Jens,

no, I used the seqs.fna output file from the split_libraries.

Here's the command I used:
denoise_wrapper.py -i NvesL.sff.txt -f split_libraries/seqs.fna -o
denoised -m mapping.txt

The denoiser.log looks like this:
Denoiser version: 1.3.0
SFF file: NvesL.sff.txt
Fasta file: split_libraries/seqs.fna
Preprocess dir: None
Primer sequence: GAGTTTGATCNTGGCTCAG
Running on cluster: False
Num CPUs: 1
Squeeze Seqs: True
tmpdir: denoised
percent_id threshold: 0.97
Minimal sequence coverage for first phase: 1
Low cut-off: 3.75
High cut-off: 4.50
Error profile: /software/qiime-1.3.0-release/lib/qiime/support_files/
denoiser/Data/FLX_error_profile.dat
Maximal number of iteration: None

Cheers,
Martin



On Sep 6, 5:56 pm, Jens Reeder <jens.ree...@gmail.com> wrote:
> HiMartin

>
> Can you post the command that you used for denoise_wrapper.
> It looks like you passed in the wrong fasta file. It must be the
> output fasta file from split_libraries.
>
> Jens
>

[Jens Reeder]

Have you checked that the seqs.fna looks okay?
E.g. do a

$ head split_libraries/seqs.fna
and
$ tail split_libraries/seqs.fna

Jens

[Martin]

Yes, the seqs.fna looks fine to me. Here's the beginning and end of
the file (sequences abbreviated):

>NvesL_1 NvesL::G6ONMY302ESC1T orig_bc=AGTCCCAA new_bc=AGTCCCAA bc_diffs=0
ATTGAACGCTGGCGGCAGGCCTAACACATG…
>NvesL_2 NvesL::G6ONMY302C1ESK orig_bc=AGTCCCAA new_bc=AGTCCCAA bc_diffs=0
ATTGAACGCTGGCGGCAGGCCTAACACATG…
>NvesL_3 NvesL::G6ONMY302C1E37 orig_bc=AGTCCCAA new_bc=AGTCCCAA bc_diffs=0
ATTGAACGCTGGCGGTATGCTTAACACATGC…
...
>NvesP_10197 NvesP::G68C4CI01B8K6S orig_bc=AGTCGTCA new_bc=AGTCGTCA bc_diffs=0
ATTGAACGCTGGCGGCATGCCTTACACATGC…
>NvesP_10198 NvesP::G68C4CI01BXY48 orig_bc=AGTCGTCA new_bc=AGTCGTCA bc_diffs=0
GACGAACGCTGGCGGCGTGCTTAACACATGC…
>NvesP_10199 NvesP::G68C4CI01AYGCO orig_bc=AGTCGTCA new_bc=AGTCGTCA bc_diffs=0
AGTGAACGCTGGCGGTAGGCCTAACACATGC…

Martin

On Sep 7, 5:47 pm, Jens Reeder <jens.ree...@gmail.com> wrote:
> Have you checked that the seqs.fna looks okay?
> E.g. do a
>
> $ head split_libraries/seqs.fna
> and
> $ tail split_libraries/seqs.fna
>
> Jens
>

[Martin]

Hi,

I found an earlier post on problems with updating to the 1.3.0 version
of QIIME, and I updated the qiime_config file according to the
suggestions, but still the denoise_wrapper and the ampliconnoise
scripts don't work for me.

Any other suggestions how to solve the problem?

Thanks,
Martin

[Antonio González Peña]

Hi Martin,

Can you send the output of print_qiime_config -t?

Thanks

--
Antonio González Peña
Research Assistant, Knight Lab
University of Colorado at Boulder
https://chem.colorado.edu/knightgroup/

[Jens Reeder]

Martin,

Sorry for not answering earlier, but I overlooked your message.

Your fasta headers in the seqs.fna file are not compatible with Qiime,
due to the extra NvesL:: in front of each flowgram identifer (the
G6ONM... string). Do you have any ideas where they came from. Where
they in your initial fasta file you got from 454. How about the
sff.txt file?

Jens

[Martin]

Hi Jens,

they're not in the sff.txt file, but in the original fasta file
supplied from the sequencing company. I deleted the NvesL:: part, and
denoise-wrapper runs now! Thanks a lot!
Ampliconnoise is still not running though, because it takes a
different input file (the sff.txt), which apparently doesn't have the
same problem (it doesn't contain the NvesL:: string). Do you have any
suggestions on that as well?

btw, which of the two denoising scripts would you recommend to use?
Are they comparable, or is one procedure clearly superior?

Thanks!
Martin

On Sep 13, 6:21 pm, Jens Reeder <jens.ree...@gmail.com> wrote:
> Martin,
>
> Sorry for not answering earlier, but I overlooked your message.
>
> Your fasta headers in the seqs.fna file are not compatible with Qiime,
> due to the extra NvesL:: in front of each flowgram identifer (the
> G6ONM... string). Do you have any ideas where they came from. Where
> they in your initial fasta file you got from 454. How about the
> sff.txt file?
>
> Jens
>

[Jens Reeder]

Hi Martin

I don't have suggestions on why AmpliconNoise failed. There are a
couple of other threads here in the forum that might address the same
problem. Maybe you can try if any of those relate to your problem.

For a comparison of AmpliconNoise and denoiser I refer you to Chris
Quince's recent(?) paper, which has a lot of comparisons.
In a nutshell, the outcome was that AmpliconNoise is more accurate
than denoiser, but a lot more expensive to run.
So it probably depends on what your constraints are, what the best choice is.

Jens