Error running "single sample" split_libraries_fastq.py

[Maria Fernanda Campa]

Hello,

I am processing data analyses for a previous MiSeq run, and one sample did not have enough sequencing depth. To correct this, we resequenced this sample, with a bunch of other unrelated sample. To facilitate data analysis, as I am only interested in that sample, the Miseq was set-up to provide demultiplex samples. 

I joined paired ends using the join_paired_ends.py script: join_paired_ends.py -f *R1_001.fastq.gz -r *R2_001.fastq.gz -o cosm8_oilsCT2_assembled -b *I1_001.fastq.gz

The output is attached here. 

Then I went forward with the split_libraries_fastq.py to get my seqs.fna file, so I can chimera check, pick OTUs, and then merge with my previous run to rarefy to the same depth. 

However, I keep getting an error with the split_libraries_fastq.py. I have never run data analysis on a single sample and I am curious if that could be messing up with something.

I tried this: split_libraries_fastq.py -i fastqjoin.join.fastq -b fastqjoin.join_barcodes.fastq -o split_lib_q20/ -m SampleSheet_Henry.txt -q 19

and then this: split_libraries_fastq.py -i fastqjoin.join.fastq -b fastqjoin.join_barcodes.fastq -o split_lib_q20/ --barcode_type 'not-barcoded' --sample_ids BP-18  -q 19

And they both gave the same error: RuntimeWarning: Mean of empty slice.

The log says this:

Input file paths

Sequence read filepath: fastqjoin.join.fastq (md5: ed6eeb5b4a6e9a2edbec4eac07fb3839)

Quality filter results

Total number of input sequences: 89736

Barcode not in mapping file: 0

Read too short after quality truncation: 12007

Count of N characters exceeds limit: 77729

Illumina quality digit = 0: 0

Barcode errors exceed max: 0

Result summary (after quality filtering)

Median sequence length: nan

BP-18 0

Total number seqs written 0

I attached the sample sheet, and this link: https://drive.google.com/drive/folders/0Bwr_-wZl6ZhERGZFWWN5bG0xcUE?usp=sharing  has all the data outputs. Thank you!!

Maria Fernanda Campa

---

[Daniel McDonald]

Hi Maria,

Does split libraries complete when just using the forward reads? Or, does it complete if -q 19 is removed? I'm not seeing anything obviously wrong

Best,

Daniel

[Maria Fernanda Campa]

Hi Daniel,

I tried it by removing -q 19 and I got the same error. I didn't try just the forward read because the fastqjoin.join.fastq and fastqjoin.join_barcades.fastq (attached in the google drive) seemed to be working fine.

mcampa@atipa:~/BP_Cosm11/Com8/cosm8_oilsCT2_assembled$  qiime > head fastqjoin.join_barcodes.fastq

@M02171:112:000000000-AUPYK:1:1101:16356:1464 1:N:0:26

ATGTGCACGACT

+

BCCBCFFFCCDC

@M02171:112:000000000-AUPYK:1:1101:16363:1482 1:N:0:26

ATGTGCACGACT

+

3A3AAFFFAADB

@M02171:112:000000000-AUPYK:1:1101:17298:1519 1:N:0:26

ATGTGCACGACT

and

mcampa@atipa:~/BP_Cosm11/Com8/cosm8_oilsCT2_assembled$  qiime > head fastqjoin.join_barcodes.fastq

@M02171:112:000000000-AUPYK:1:1101:16356:1464 1:N:0:26

ATGTGCACGACT

+

BCCBCFFFCCDC

@M02171:112:000000000-AUPYK:1:1101:16363:1482 1:N:0:26

ATGTGCACGACT

+

3A3AAFFFAADB

@M02171:112:000000000-AUPYK:1:1101:17298:1519 1:N:0:26

ATGTGCACGACT

mcampa@atipa:~/BP_Cosm11/Com8/cosm8_oilsCT2_assembled$  qiime > head fastqjoin.join.fastq

@M02171:112:000000000-AUPYK:1:1101:16356:1464 1:N:0:26

TACTAAAGGTGCAAGCGTTAATCGGAATTACTGGGCGTAAAGCGCGCGTAGGTGGTTTGTTAAGTTGGATGTGAAAGCCCAGGGCTCAACCTTGGAACTGCATTCAAAACTGACTCACTAGAGTACGAGAGAGGTTAGTGGAATTTCCTGTGTAGCGGTGAANTGCGTANAGATGGGNNGGAACACCNGTGGCNNNGNCGACTGCCTGGNCTGATNCTNACNCTGAGGTGCGANAGCGTGGGGAGCAAACAGG

+

CCCCCFFFFFFFGGGGGGGGGGHGGGGGHHHHHFHHGGGGHHHGGGGGEGGGEGGGGHFGHHHGFGFHHHHHHHHHGHHHHHFEFHFGHHHHHHHGHHHHHHHHGHHHHHHGFHHHHHHHHHHHHGHHHHHHHGGGHGGF##HHHHHHHHHGGGEE1FFF??#CGFE??#HHGFFBB##FHEFFB>>#FCBBB###B#GGHGHGGEEBB#FFFBA#BB#AA#HHHHGGGFFAA#GCGGGGFFFFFFFFAAA>>

@M02171:112:000000000-AUPYK:1:1101:16363:1482 1:N:0:26

TACTAAAGGTGCAAGCGTTAATCGGAATTACTGGGCGTAAAGCGCGCGTAGGTGGTTTGTTAAGTTGGATGTGAAAGCCCAGGGCTCAACCTTGGAACTGCATTCAAAACTGACTCACTAGAGTACGAGAGAGGTTAGTGGAATTTCCTGTGTAGCGGTGAANTGCGTANAGATGGGNNGGAACACCNGTGGCNNNGNCGACTGCCTGGNCTGATNCTNACNCTGAGGTGCGANAGCGTGGGGAGCAAACAGG

+

AAAA1D1DFFFFGGGGGGFGGGHGGECGHHFHGHHGGGG/BGHGGGGGGAGGGHGGGEGHHGFHHHHHGHHHGH2GBGHHHHGGGGHHHHHHGHHHHHHHHHHHHHHHHHHGHHHBHHHHHHHHHGHHHGHHGGCGFFHH##HHHHHHHHGE/GGGHGFF??#EFFF??#FFFFFBB##HHE?F?1B#FFBBB###B#EHHGGHGFFBB#GGFBB#AA#BB#HHHHGGFFFAA#GGGGCGGFFFFFFFBBBAA

@M02171:112:000000000-AUPYK:1:1101:17298:1519 1:N:0:26

TACTAAAGGTGCAAGCGTTAATCGGAATTACTGGGCGTAAAGCGCGCGTAGGTGGTTTGTTAAGTTGGATGTGAAAGCCCAGGGCTCAACCTTGGAACTGCATTCAAAACTGACTCACTAGAGTACGAGAGAGGTTAGTGGAATTTCCTGTGTAGCGGTGAANTGCGTANAGATGGGNNGGAACACCNGTGGCNNNGNCGACTGACTGGNTCGATNCTNACNCTGAGGTGCGANAGCGTGGGGAGCAAACAGG

I am stuck, and would appreciate any suggestions to move forward.

Thank you,

[Daniel McDonald]

Hi Maria,

It doesn't seem like fastqjoin.join.fastq and fastqjoin.join_barcades.fastq are working fine as all of the sequences are being dropped? Given the contents of the head, it appears there are quite a few ambiguous bases, and by eyeball, they all seem toward the 3' end. Typically, the reverse read is much worse quality, and while you can join the reads, it comes at a large trade off as you either need to toss out more sequence due to quality, or reduce your tolerance for noise. As such, you may be better off just using the forward reads. If you'd like to tolerate additional noise, I recommend looking at changing the values associated with the number of ambiguous bases allowed, as well as changing the read length required following truncation (as split libraries will truncate when the quality begins to drop below threshold).

Best,

Daniel

[Maria Fernanda Campa]

Hi Daniel, 

You are correct, I glossed over the reverse read and did not notice the quality drop. I did as you suggested by using split_libraries_fastq.py with only the forward read, and then I check and filter out chimeras. Everything seemed to have worked until I ran the pick_open_reference_otus.py. It stopped half way, step 1 and 4, worked but the error was raised for step 2.

I ran this:

 mcampa@atipa:~/BP_Cosm11/Com8/split_lib_q20$  qiime > pick_open_reference_otus.py -i seqs_chimera_filtered -o otus -r /usr/share/qiime/data/gg_13_8_otus/rep_set/97_otus.fasta -m uclust -p /home/mcampa/qiime_parameters.txt -s 0.1 -a -O 20

This is the error that I got:

*** ERROR RAISED DURING STEP: Add taxa to OTU table

Command run was:

 biom add-metadata -i otus/otu_table_mc2.biom --observation-metadata-fp otus/rdp_assigned_taxonomy/rep_set_tax_assignments.txt -o otus/otu_table_mc2_w_tax.biom --sc-separated taxonomy --observation-header OTUID,taxonomy

Command returned exit status: 2

Stdout:

Stderr

/usr/bin/biom: 22: export: 4.0/dist-packages: bad variable name

What do you think? I included the log file and qiime parameter I used, I can't figure out what went wrong I use this same protocol all the time, but I have never done it in for a single sample. 

Thank you!

[Daniel McDonald]

Hi Maria,

I apologize for the delay -- this got lost in my inbox. It appears that your BIOM installation is bad. Can you send the output of:

$ biom show-install-info

And do you know how BIOM was installed? 

Thanks,

Daniel

[Maria Fernanda Campa]

Hi Daniel,

Thanks for your response. 

This is the output:

mcampa@atipa:~$  qiime > biom show-install-info

/usr/bin/biom: 22: export: 4.0/dist-packages: bad variable name

It seems there is an error with that. We have Qiime in our server through biolinux. To the best of my knowledge no one has changed or reinstalled BIOM. Should I delete an re-install?

Thank you!

Maria

[Daniel McDonald]

Hi Maria,

Yes. My guess is that someone manually modified a library file as that error is very odd given a brief google search.

Best,

Daniel