Denoising of 454 Data Sets

[Kirsi Hyytiäinen]

Hello QIIME developers

Can you tell me what is wrong with my mapping file?

Original file called Marine_map1.txt
#SampleID BarcodeSequence LinkerPrimerSequence ReversePrimer
Description
HDIG4LN01D9D53 AGAGTTTGATCMTGGCTCAG GTATTACCGCGGCTGCTG
Spring08Storfjarden
HDIG4LN01EL066 AGAGTTTGATCMTGGCTCAG GTATTACCGCGGCTGCTG
Spring08Storfjarden

exact command
check_id_map.py -m Marine_map1.txt -o mapping1_output -v

error log file
#Listed locations of errors/warnings in row/column format have an
index that is
#in reference to the beginning of the sample IDs and metadata.
#Location (0,0) is the first SampleID in a given data set.
ERRORS-------------------
0: DupChecker 'BarcodeSequence' found the following possible
duplicates. If these metadata should have the same name, please
correct.:
Group Original names
,
Row, column for all possible duplicate descriptions:
Location (row, column): 0,1
Location (row, column): 1,1


WARNINGS -------------------
0: Missing barcode. Location (row, column): 0,1
1: Missing barcode. Location (row, column): 1,1


System information
==================
Platform: linux2
Python version: 2.7.1 (r271:86832, Dec 14 2011, 00:47:21) [GCC
4.4.3]
Python executable: /software/python-2.7.1-release/bin/python

Dependency versions
===================
PyCogent version: 1.5.1
NumPy version: 1.5.1
matplotlib version: 1.1.0
QIIME library version: 1.4.0
QIIME script version: 1.4.0
PyNAST version (if installed): 1.1
RDP Classifier version (if installed): rdp_classifier-2.2.jar

QIIME config values
===================
blastmat_dir: /software/blast-2.2.22-release/data
topiaryexplorer_project_dir: None
pynast_template_alignment_fp: /software/
core_set_aligned.fasta.imputed
cluster_jobs_fp: /software/qiime-1.4.0-release/bin/
start_parallel_jobs.py
pynast_template_alignment_blastdb: None
assign_taxonomy_reference_seqs_fp: /software/gg_otus-4feb2011-release/
rep_set/gg_97_otus_4feb2011.fasta
torque_queue: friendlyq
template_alignment_lanemask_fp: /software/lanemask_in_1s_and_0s
jobs_to_start: 1
cloud_environment: False
qiime_scripts_dir: /software/qiime-1.4.0-release/bin
denoiser_min_per_core: 50
working_dir: /tmp/
python_exe_fp: /software/python-2.7.1-release/bin/
python
temp_dir: /tmp/
blastall_fp: /software/blast-2.2.22-release/bin/
blastall
seconds_to_sleep: 60
assign_taxonomy_id_to_taxonomy_fp: /software/gg_otus-4feb2011-release/
taxonomies/greengenes_tax_rdp_train.txt


Thanks Kirsi

[Jesse Stombaugh]

Hello Kirsi,

The error is produced due 2 of your samples having the same barcode, which means that when you demultiplex the samples, you will not know which sample each sequence goes with.

-Jesse

2012/1/25 Kirsi Hyytiäinen <kirs...@gmail.com>

--
Jesse Stombaugh, Ph.D.
Research Associate
University of Colorado, Boulder

[Kirsi Hyytiäinen]

Heillo Jesse,

Thanks you for your answer, but now I have another problem.

I have 12 samples, each sample contains about 14 000 sequences, so how do I get barcodes for them. All samples have tag sequences (sample 1 has ACGACG, sample 2 has TCGTCG etc.) before forward primers, other than that I do not know how to differentiate them.

Do you have any suggestions?

Thanks Kirsi

[Jesse Stombaugh]

Hello Kirsi,

Have your sequences already been demultiplexed, since you know the number of seqs/sample. If the samples are already demultiplexed, are you trying to remove the primers and barcodes? Which script in QIIME are you currently stuck on? 

[Kirsi Hyytiäinen]

Hello Jesse,

First I run a python script that calculates the length of the sequences and the number of sequences.

I installed QIIME VirtualBox and run the example data set. It was cool.

I tried to follow QIIME instructions: Denoising of 454 Data Sets with my own data,

 and now I am stuck, because I do not understand what is a barcode or what to do exactly.

Thanks fo ryour  help

Kirsi

[Jesse Stombaugh]

Hello Kirsi,

Well, the first question I should ask is what sequencing technology was used and did you barcode your sequences? Can you also explain what type of sequences you are working with (i.e. 16S, metagenomic, etc.)? If you sent your samples to a sequencing facility, where they performed the barcoding step, I suggest you email them and politely ask for the barcodes that were used for each sample, otherwise the person in your lab that performed DNA extraction/amplification, etc. should have that information.  It is also possible they already assigned each sequence to a sample, which means you may not need to run the split_libraries.py script.

[Kirsi Hyytiäinen]

Hello Jesse,

I am a bioinformatician, I just got this data to be analyzed. 

The data is processed by 454 pyrosequencing machine in a company and there are 3 files: one fasta file (.seq), one quality file (.qual) and one .sff file.

This is what I know: There are 12 samples from different years and locations from a sea, so they are marine 16S DNA sequences (metagenomics).

Each sample has it's own tag sequence, but an individual sequence has no specific codes (barcodes?). Each sequence in a single file starts with the same tag sequence (Sample1 tag1 ACGACG, Sample2 tag2 TCGTCG etc.)

followed by forward primer sequence +16S DNA sequences + reverse primer sequence+adaptor sequence.

And I need to know What differences are there between the samples, What is the taxonomic content of each sample, What is the functional content etc.

[Jesse Stombaugh]

Hello,

Thank you for the clarification. The tag is the barcode, so there should be a unique tag for each sample. If you put those tags in the BarcodeSequence column of the mapping file, then you can demultiplex the samples using the split-libraries.py script. This should solve any issues with the denoising tutorial as well.

Let me know if you have any trouble,

[Kirsi Hyytiäinen]

Hello Jesse,

I just cannot get these barcodes right.  As far as I understand I should have a unique barcode for every sequence, but I have a barcode for every sample (each sample consisting of 14000 sequences).

Thanks for your help

Kirsi

command

check_id_map.py -m test.txt -o testout

#Listed locations of errors/warnings in row/column format have an index that is 

#in reference to the beginning of the sample IDs and metadata.

#Location (0,0) is the first SampleID in a given data set.

ERRORS-------------------

0: DupChecker 'BarcodeSequence' found the following possible duplicates. If these metadata should have the same name, please correct.:

Group Original names

AGCAGC AGCAGC, AGCAGC

TACAGC TACAGC, TACAGC

Row, column for all possible duplicate descriptions:

Location (row, column): 0,1

Location (row, column): 1,1

Location (row, column): 2,1

Location (row, column): 3,1

[Antonio González Peña]

Hi Kirsi,

You need a unique barcode for each sample for check_id_map.py to pass,
then when you run split_libraries.py this script will assign unique
ids to each sequence.

Hope this helps.

2012/1/26 Kirsi Hyytiäinen <kirs...@gmail.com>:

--
Antonio González Peña
Research Assistant, Knight Lab
University of Colorado at Boulder
https://chem.colorado.edu/knightgroup/

[Kirsi Hyytiäinen]

Hi Antonio

I have 12 files, each of them contains about 14 000 sequences. Each file has a unique barcode but individual sequences have no barcode.

How can I generate a unique barcode for each sequence?

Thanks 

Kirsi

2012/1/26 Antonio González Peña <antg...@gmail.com>

[Tony Walters]

Hello Kirsi,

Just to clarify the situation:

You have 12 total pairs of .fna and .qual files.  They've already been separated into these files by the original 12 samples, and the barcode has been removed (was this removal done by the sequencing company?  Was this done in Lubbock, Texas by any chance?).

Because the barcode has been removed, as well as the corresponding quality scores in the .qual files, it makes any attempt to manually reappend the barcodes impossible.

One approach would be to create 12 separate mapping files, one for each sample (you also want a combined mapping file with all the samples in it for downstream analyses, just use the -b option with check_id_map to make sure the mapping file is okay).

So for HDIG4LN01D9D53, you would just have something like this:

Original file called Marine_map1.txt
#SampleID       BarcodeSequence LinkerPrimerSequence    ReversePrimer Description

Description
HDIG4LN01D9D53          AGAGTTTGATCMTGGCTCAG    GTATTACCGCGGCTGCTG Marine_Sample1

with the BarcodeSequence field empty.

when you run check_id_map.py on each file, include the -b option.

Then you will need to run split_libraries with the -b 0 option on each file, and a different output directory for each run (-o option).

Finally, you can concatenate all of the results by using the following command:

cat path to first seqs.fna output file path to second seqs.fna output file .... path to twelfth seqs.fna output file > combined_seqs.fna

There may be an easier way, but it would depend upon there being some sort of identifier in the fasta labels that could be used to help assign the sequences to particular SampleIDs.

You can get the first 10 lines of a fasta file by typing:

head path_to_fasta > sample_seqs1.fna

If you do this for a couple of your files, we could look at the fasta labels to see if there are any useful sample-specific data in the labels that we can use to demultiplex in a single step.

Best regards,

Tony Walters

2012/1/26 Kirsi Hyytiäinen <kirs...@gmail.com>

[Lily]

Hello Tony,

 

I faced the similar situation with Kirsi. The barcodes in my pyrosequencing results were already movedwere.I already did split by forward primer as you said. And I want to know how can I do for denoiser. Because when I gave the commond:

denoise_wrapper.py -v -i MID1.sff.txt -f seqs.fna -o denoised -m W4_map5_BAC.txt --titanium 

 

The names of reads in the outputfile "denoised/denoised_seqs.fasta" are not as the Sample ID of mapping file and imput file. I do not why it came out like that. I did all the Shared_Folder. Is it the reason?

 

ATTACHMENT:

The outputfile denoised/denoised_seqs.fasta is as:

>HI85TKX04H1BLA | cluster size: 2905
CGTGGCTCAGAGTGAACGCTGGCGGCGT...........
>HI85TKX04ISJOR | cluster size: 1159

CGTGGCTCAGAGTGAACGCTGGCGGCGT...........

The names of reads are not as the Sample ID of mapping file and imput file:

>W4.Bac_1 HI85TKX04IWJNF orig_bc= new_bc= bc_diffs=0
AGTGAACGCTGGCGGCGTGCTTAACACATGCAAGTCGAACGAG.........
>W4.Bac_2 HI85TKX04JXOSX orig_bc= new_bc= bc_diffs=0
AGTGAACGCTGGCGGCGTGCTTAACACATGCAAGTCGAACGAG..........

 

My output log file is :

Denoiser version: 1.5.0
SFF file: MID1.txt
Fasta file: split_library_output_W4_BAC/seqs.fna
Preprocess dir: None
Primer sequence: GAGTTTGATCNTGGCTCAG
Running on cluster: False
Num CPUs: 1
Squeeze Seqs: True
tmpdir: split_library_output_W4_BAC/denoised/
percent_id threshold: 0.97
Minimal sequence coverage for first phase: 1
Low cut-off: 4.00
High cut-off: 5.00
Error profile: /home/qiime/qiime_software/qiime-1.5.0-release/lib/qiime/support_files/denoiser/Data/Titanium_error_profile.dat
Maximal number of iteration: None
......

 

My mapping file is :

#SampleID BarcodeSequence LinkerPrimerSequence ReversePrimer Description

W4.Bac  GAGTTTGATCNTGGCTCAG GTNTTACNGCGGCKGCTG Bacteria

 

Hope to hear from you.