what sequence identifiers should be for pick_de_novo_otus.py

[Tonja Rand]

I have paired-end reads. Due to some library problems, I demultiplexed and cut the adapter/linker sequences by my own. For each sample I have a folder with the corresponding forward and reverse reads. Firstly, I decided not to merge the forward and reverse reads and just have a look what clusters I get by running pick_de_novo_otus.py only on forward reads.

After running the script it shows me the following error:

pick_de_novo_otus.py  -i sample1/forward_fasta/f.fna -o sample1/forward_fasta/otus

ValueError: An empty fasta file was provided. Did the alignment complete sucessfully? Did PyNAST discard all sequences due to too-stringent minimum length or minimum percent ID settings?

All sequences are around 260 bp. I think this error is due to the fact that the sequences identifiers in the file is not correct as I did not run split_libraries.py. In the documentation I found following: "The sequence identifiers in this file should be of the form <sample_id>_<unique_seq_id>"

In my sample1/forward_fasta/f.fna there is not sample id as for each sample I have its own folder. Is it a requirement to have a sample id in the identifier? If yes, could it be the reason why I do not get the clusters?

/forward_fasta/f.fna looks as follows:

@M04070:53:000000000-AUCFU:1:1101:21310:2043 1:N:0:102

TATTGGAGCTGGAATTACCGCGGCTGCTGGCACTAGACTTGCCCTCCAATGGATCTTCGTTAAAGGATTTAAAGTGGACTCATTCCAATTACAGGACCTCGAAAGAGTCCTGTATTGGTATGTTTCATCACTACCTCCTTGGGTCGGGAGTGGGTAATTTGCATGCCTGCTGCCTTCCTTGAATGTGGTATCCATTTCTCAGGCTCCCTCTACAGTATTCAACCCTGATTCCCCATCACCCGTGGTCACCATGGTAGGCACAG

+

GGGGGGGGGGGGGGGGEFGEGDCFGCEG<CC,FFCE9EFF<FFGGGG8<EEE9FGGGG@FGF7<A98EEGFC9CC,5,EFGCFFFE9EFGCEFCFDFGFGCC,+><EFGFG9FFFGD9F,CFFGFGCFG9FGDFFGGGGGC8AFF7>C+CCCCGGE,EFG9F9E9DFF9FGBFGGGFGGG7,,>>DDFGC,5DCFC@:@,,,*0;CDGG<F8C>8++=C+4+;;>G54ACFACA40=36CDB8(0>A;AE4?23@9*4<<:4(

@M04070:53:000000000-AUCFU:1:1101:19674:2260 1:N:0:102

TATTGGAGCTGGAATTACCGCGGCTGCTGGCACCAGACTTGCCCTCCAATTGTTCCTCGTTAAGGGATTTAAATTGTACTCATTCCAATTACAAGACCCAAAAGAGCCCTGTATCAGTATTTATTGTCACTACCTCCCCGTGTCGGGATTGGGTAATTTGCGCGCCTGCTGCCTTCCTTGGATGTGGTAGCCGTTTCTCAGGCTCCCTCTCCGGAATAGAACCCTAATTCCCCGTTACCCGTTGATACCATGGTAGGCCACTAT

+

GGGGGGGGGGGGGGGGGGGGGGGGGGDGFFGGGGGGGGGGFGGEGGGGGGGGGGGGGGGGGGGGGG?FFGGGGGGFGGGGGGGGGGGGGGGGFGCGGGGGCFFGFGGGGGGGGGGGGGGGAE?FGGGGGFGGGGFGGGGGGGFGEG:EFGGCFEG8EFAFGGGGGGGGGGGGGG7EGGGGGDDFCFGFGGGGGGGCFGGGCFD*;ECFFFGGGG*85=>FGGGAFGFF=@=?GFEFFFFGB>)8FFFFF???<AF4>?8??FB)

@M04070:53:000000000-AUCFU:1:1101:22694:2710 1:N:0:102

TATTGGAGCTGGAATTACCGCGGCTGCTGGCACCAGACTTGCCCTCCAATGGATCCTCGTTAAGGGATTTAGATTGTACTCATTCCAATTACCAGACTCATAGAGCCCGGTATTGTTATTTATTGTCACTACCTCCCCGTGTCAGGATTGGGTAATTTGCGCGCCTGCTGCCTTCCTTGGATGTGGTAGCCGTTTCTCAGGCTCCCTCTCCGGAATCGAACCCTTATTCCCCGTTACCCGTTGAAACCATGGTAGGCCTCTAT

Attached is the log file from dynast_aligned_seqs

[Daniel McDonald]

Hi Tonja,

It is a requirement to have the data in <sample_id>_<unique ID> format. It should be possible to add the sample IDs using add_qiime_labels.py (http://qiime.org/scripts/add_qiime_labels.html).

Best,

Daniel

[Tonja Rand]

Hallo Daniel,

Thank you for your replay. I was able to add a sample id. However, it still throws me the same ValueError.

As I said, the reads are around 260 bp, there are 14730 (only forward) reads. I have doubts that there is no any subset of reads that could build a cluster. I guess I am doing something totally wrong.

add_qiime_labels.py -m mapping_radtags.txt -i forward_fasta -c InputFileName -o forward_fasta/combined_sample_fasta

pick_de_novo_otus.py  -i forward_fasta/combined_sample_fasta/combined_seqs.fna -o forward_fasta/combined_sample_fasta/otus

In combined_seqs.fna I have only one sample. I want to cluster only those reads which are within one sample. Is it the right strategy then? Or should there be different samples in my combined_seqs.fna ?

My combined_seqs.fna:

>Sample.1_0 M04070:53:000000000-AUCFU:1:1101:7469:2101

AGTCATATGCTTGTCTCAAAGATTAAGCCATGCATGTCTAAGTATACGCGATATACAGTGAAACTGCGAATGGCTCATTAAATCAGTTATCGTTTATTTGATAGTACTTACTACATGGATATCCGTGGTAATTCTAGAGCTAATACATGCCTTTCTTGCATTTATTATATAAACCGACGTTTTGGCGATTCATAATAACTTGCGGACTCCTGCGTGGGCGGTTCATTCACATTTCTGCCCTATCTACTCTCGTTTGTAGGCTATCGTC

>Sample.1_1 M04070:53:000000000-AUCFU:1:1101:14446:2285

AGTCATATGCTTGTCTCAAAGATTAAGCCATGCATGTCTAAGTATACGCGATATACAGTGAAGCTGCGAATGGCTCATTAAATCAGTTATCGTTTATTTGATAGTACTTACTACATGGATATCCGTGGTAATTCTAGAGCTAATACATGCCTTTCTTGCATTTATTAGATAAACCGACGTTTTGGCGATTCATAATAACTTGCGGACCCCTGCGGGGGCGGTTCATTCAAATTTCTGCCCTATCAACTTTCGCTGGTAGGATAGAGGC

>Sample.1_2 M04070:53:000000000-AUCFU:1:1101:17267:2307

AGTCATATGCTTGTCTCAAAGATTAAGCCATGCATGTCTAAGTATACGCGATATACAGTGAAACTGCGAATGGCTCATTAAATCAGTTATCGTTTATTTGATAGTACTTACTACATGGATATCCGTGGTAATTCTAGAGCTAATACATGCCTTTCTTGCATTTATTAGATAAACCGACGTTTTGGCGATTCATAATAACTTTCGGACCCCTGCGGGGGCGGTTAATTCAAATTTCGGCCCTATCAACTTTCGATGGTAGGGTAGAGGCC

>Sample.1_3 M04070:53:000000000-AUCFU:1:1101:15923:2309

AGTCATATGCTTGTCTCAAAGATTAAGCCATGCATGTCTAAGTATACGCGATATACAGTGAAACTGCGAATGGCTCATTAAATCAGTTATCGTTTATTTGATAGTACTTACTACATGGATATCCGTGGTAATTCTAGAGCTAATACATGCCTTTCTTGCATTTATTAGATAAACCGACGTTTTGGCGATTCATAATAACTTGCGGACCCCTTCGGGGGCGGTTCATTCAAATTGCTGCCCTATCCACTTTCGATGGGAGGATAGAGGC

>Sample.1_4 M04070:53:000000000-AUCFU:1:1101:20872:2345

AGTCATATGCTTGTCTCAAAGATTAAGCCATGCATGTCTAAGTATAAGCAATTTATACAGTGAAACTGCGAATGGCTCATTAAATCAGTTATCGTTTATTTGATAGTTCCTTTACTACATGGTATAACTGTGGTAATTCTAGAGCTAATACATGCTTAAAATCTCGACCCTTTGGAAGAGATGTATTTATTAGATAAAAAATCAATGTCTTCGGACTCTTTGATGATTCATAAGAACTTTTCGAATCGCCTGGCCTTGTGCTGGCGATG

File otus/pynast_aligned_seqs/combined_seqs_rep_set_log.txt contains following:

candidate sequence ID candidate nucleotide count errors template ID BLAST percent identity to template candidate nucleotide count post-NAST

denovo0 Sample.1_12623 269 No search results.

denovo1 Sample.1_12624 269 No search results.

denovo10 Sample.1_1266 269 No search results.

denovo100 Sample.1_12106 268 No search results.

denovo1000 Sample.1_10022 272 No search results.

denovo1001 Sample.1_10531 268 No search results.

denovo1002 Sample.1_10024 269 No search results.

denovo1003 Sample.1_10537 268 No search results.

denovo1004 Sample.1_10026 268 No search results.

denovo1005 Sample.1_10027 269 No search results.

[Daniel McDonald]

Hi Tonja,

Thank you for the additional details, this is quite useful. I BLASTed a few of the sequences and the results suggest the data are 18S. If so, that would explain why pynast is not aligning as the default template it is using is composed of 16S sequences. QIIME compatible files from SILVA (which includes 18S) can be found here:

https://www.arb-silva.de/download/archive/qiime/

My guess is that will allow for some of the reads to align.

Note, depending on what you want to subsequently do, you do not need an alignment. The primary purpose of the alignment is for phylogenetic reconstruction to leverage phylogeny-aware diversity metrics. So for instance, if you just want to do de novo OTU picking, you should just be able to run pick_otus.py directly (details here). I believe the output of that command would need to be run through make_otu_table.py. Doing the OTU picking "manually" like this will avoid the alignment, but it will also avoid things like taxonomy assignment which would also need to be done manually.

Best,

Daniel

Best,

Daniel