Error running "single sample" split_libraries_fastq.py

60 views
Skip to first unread message

Maria Fernanda Campa

unread,
Nov 11, 2016, 12:38:18 PM11/11/16
to Qiime 1 Forum
Hello,

I am processing data analyses for a previous MiSeq run, and one sample did not have enough sequencing depth. To correct this, we resequenced this sample, with a bunch of other unrelated sample. To facilitate data analysis, as I am only interested in that sample, the Miseq was set-up to provide demultiplex samples. 

I joined paired ends using the join_paired_ends.py script: join_paired_ends.py -f *R1_001.fastq.gz -r *R2_001.fastq.gz -o cosm8_oilsCT2_assembled -b *I1_001.fastq.gz

The output is attached here. 

Then I went forward with the split_libraries_fastq.py to get my seqs.fna file, so I can chimera check, pick OTUs, and then merge with my previous run to rarefy to the same depth. 

However, I keep getting an error with the split_libraries_fastq.py. I have never run data analysis on a single sample and I am curious if that could be messing up with something.

I tried this: split_libraries_fastq.py -i fastqjoin.join.fastq -b fastqjoin.join_barcodes.fastq -o split_lib_q20/ -m SampleSheet_Henry.txt -q 19

and then this: split_libraries_fastq.py -i fastqjoin.join.fastq -b fastqjoin.join_barcodes.fastq -o split_lib_q20/ --barcode_type 'not-barcoded' --sample_ids BP-18  -q 19

And they both gave the same error: RuntimeWarning: Mean of empty slice.

The log says this:

Input file paths
Sequence read filepath: fastqjoin.join.fastq (md5: ed6eeb5b4a6e9a2edbec4eac07fb3839)
Quality filter results
Total number of input sequences: 89736
Barcode not in mapping file: 0
Read too short after quality truncation: 12007
Count of N characters exceeds limit: 77729
Illumina quality digit = 0: 0
Barcode errors exceed max: 0

Result summary (after quality filtering)
Median sequence length: nan
BP-18 0

Total number seqs written 0

I attached the sample sheet, and this link: https://drive.google.com/drive/folders/0Bwr_-wZl6ZhERGZFWWN5bG0xcUE?usp=sharing  has all the data outputs. Thank you!!

Maria Fernanda Campa
---
SampleSheet_Henry.txt

Daniel McDonald

unread,
Nov 14, 2016, 12:10:27 PM11/14/16
to Qiime 1 Forum
Hi Maria,

Does split libraries complete when just using the forward reads? Or, does it complete if -q 19 is removed? I'm not seeing anything obviously wrong

Best,
Daniel

Maria Fernanda Campa

unread,
Nov 14, 2016, 2:07:14 PM11/14/16
to Qiime 1 Forum
Hi Daniel,

I tried it by removing -q 19 and I got the same error. I didn't try just the forward read because the fastqjoin.join.fastq and fastqjoin.join_barcades.fastq (attached in the google drive) seemed to be working fine.

mcampa@atipa:~/BP_Cosm11/Com8/cosm8_oilsCT2_assembled$  qiime > head fastqjoin.join_barcodes.fastq

@M02171:112:000000000-AUPYK:1:1101:16356:1464 1:N:0:26

ATGTGCACGACT

+

BCCBCFFFCCDC

@M02171:112:000000000-AUPYK:1:1101:16363:1482 1:N:0:26

ATGTGCACGACT

+

3A3AAFFFAADB

@M02171:112:000000000-AUPYK:1:1101:17298:1519 1:N:0:26

ATGTGCACGACT


and



mcampa@atipa:~/BP_Cosm11/Com8/cosm8_oilsCT2_assembled$  qiime > head fastqjoin.join_barcodes.fastq

@M02171:112:000000000-AUPYK:1:1101:16356:1464 1:N:0:26

ATGTGCACGACT

+

BCCBCFFFCCDC

@M02171:112:000000000-AUPYK:1:1101:16363:1482 1:N:0:26

ATGTGCACGACT

+

3A3AAFFFAADB

@M02171:112:000000000-AUPYK:1:1101:17298:1519 1:N:0:26

ATGTGCACGACT

mcampa@atipa:~/BP_Cosm11/Com8/cosm8_oilsCT2_assembled$  qiime > head fastqjoin.join.fastq

@M02171:112:000000000-AUPYK:1:1101:16356:1464 1:N:0:26

TACTAAAGGTGCAAGCGTTAATCGGAATTACTGGGCGTAAAGCGCGCGTAGGTGGTTTGTTAAGTTGGATGTGAAAGCCCAGGGCTCAACCTTGGAACTGCATTCAAAACTGACTCACTAGAGTACGAGAGAGGTTAGTGGAATTTCCTGTGTAGCGGTGAANTGCGTANAGATGGGNNGGAACACCNGTGGCNNNGNCGACTGCCTGGNCTGATNCTNACNCTGAGGTGCGANAGCGTGGGGAGCAAACAGG

+

CCCCCFFFFFFFGGGGGGGGGGHGGGGGHHHHHFHHGGGGHHHGGGGGEGGGEGGGGHFGHHHGFGFHHHHHHHHHGHHHHHFEFHFGHHHHHHHGHHHHHHHHGHHHHHHGFHHHHHHHHHHHHGHHHHHHHGGGHGGF##HHHHHHHHHGGGEE1FFF??#CGFE??#HHGFFBB##FHEFFB>>#FCBBB###B#GGHGHGGEEBB#FFFBA#BB#AA#HHHHGGGFFAA#GCGGGGFFFFFFFFAAA>>

@M02171:112:000000000-AUPYK:1:1101:16363:1482 1:N:0:26

TACTAAAGGTGCAAGCGTTAATCGGAATTACTGGGCGTAAAGCGCGCGTAGGTGGTTTGTTAAGTTGGATGTGAAAGCCCAGGGCTCAACCTTGGAACTGCATTCAAAACTGACTCACTAGAGTACGAGAGAGGTTAGTGGAATTTCCTGTGTAGCGGTGAANTGCGTANAGATGGGNNGGAACACCNGTGGCNNNGNCGACTGCCTGGNCTGATNCTNACNCTGAGGTGCGANAGCGTGGGGAGCAAACAGG

+

AAAA1D1DFFFFGGGGGGFGGGHGGECGHHFHGHHGGGG/BGHGGGGGGAGGGHGGGEGHHGFHHHHHGHHHGH2GBGHHHHGGGGHHHHHHGHHHHHHHHHHHHHHHHHHGHHHBHHHHHHHHHGHHHGHHGGCGFFHH##HHHHHHHHGE/GGGHGFF??#EFFF??#FFFFFBB##HHE?F?1B#FFBBB###B#EHHGGHGFFBB#GGFBB#AA#BB#HHHHGGFFFAA#GGGGCGGFFFFFFFBBBAA

@M02171:112:000000000-AUPYK:1:1101:17298:1519 1:N:0:26

TACTAAAGGTGCAAGCGTTAATCGGAATTACTGGGCGTAAAGCGCGCGTAGGTGGTTTGTTAAGTTGGATGTGAAAGCCCAGGGCTCAACCTTGGAACTGCATTCAAAACTGACTCACTAGAGTACGAGAGAGGTTAGTGGAATTTCCTGTGTAGCGGTGAANTGCGTANAGATGGGNNGGAACACCNGTGGCNNNGNCGACTGACTGGNTCGATNCTNACNCTGAGGTGCGANAGCGTGGGGAGCAAACAGG


I am stuck, and would appreciate any suggestions to move forward.

Thank you,

Daniel McDonald

unread,
Nov 15, 2016, 12:00:36 PM11/15/16
to Qiime 1 Forum
Hi Maria,

It doesn't seem like fastqjoin.join.fastq and fastqjoin.join_barcades.fastq are working fine as all of the sequences are being dropped? Given the contents of the head, it appears there are quite a few ambiguous bases, and by eyeball, they all seem toward the 3' end. Typically, the reverse read is much worse quality, and while you can join the reads, it comes at a large trade off as you either need to toss out more sequence due to quality, or reduce your tolerance for noise. As such, you may be better off just using the forward reads. If you'd like to tolerate additional noise, I recommend looking at changing the values associated with the number of ambiguous bases allowed, as well as changing the read length required following truncation (as split libraries will truncate when the quality begins to drop below threshold).

Best,
Daniel

Maria Fernanda Campa

unread,
Nov 17, 2016, 11:14:02 AM11/17/16
to Qiime 1 Forum
Hi Daniel, 

You are correct, I glossed over the reverse read and did not notice the quality drop. I did as you suggested by using split_libraries_fastq.py with only the forward read, and then I check and filter out chimeras. Everything seemed to have worked until I ran the pick_open_reference_otus.py. It stopped half way, step 1 and 4, worked but the error was raised for step 2.

I ran this:

 mcampa@atipa:~/BP_Cosm11/Com8/split_lib_q20$  qiime > pick_open_reference_otus.py -i seqs_chimera_filtered -o otus -r /usr/share/qiime/data/gg_13_8_otus/rep_set/97_otus.fasta -m uclust -p /home/mcampa/qiime_parameters.txt -s 0.1 -a -O 20

This is the error that I got:

*** ERROR RAISED DURING STEP: Add taxa to OTU table

Command run was:

 biom add-metadata -i otus/otu_table_mc2.biom --observation-metadata-fp otus/rdp_assigned_taxonomy/rep_set_tax_assignments.txt -o otus/otu_table_mc2_w_tax.biom --sc-separated taxonomy --observation-header OTUID,taxonomy

Command returned exit status: 2

Stdout:


Stderr

/usr/bin/biom: 22: export: 4.0/dist-packages: bad variable name


What do you think? I included the log file and qiime parameter I used, I can't figure out what went wrong I use this same protocol all the time, but I have never done it in for a single sample. 


Thank you!

log_20161117002009.txt
qiime_parameters.txt

Daniel McDonald

unread,
Nov 28, 2016, 12:47:08 PM11/28/16
to Qiime 1 Forum
Hi Maria,

I apologize for the delay -- this got lost in my inbox. It appears that your BIOM installation is bad. Can you send the output of:

$ biom show-install-info

And do you know how BIOM was installed? 

Thanks,
Daniel

Maria Fernanda Campa

unread,
Nov 28, 2016, 6:19:19 PM11/28/16
to Qiime 1 Forum
Hi Daniel,

Thanks for your response. 

This is the output:

mcampa@atipa:~$  qiime > biom show-install-info

/usr/bin/biom: 22: export: 4.0/dist-packages: bad variable name


It seems there is an error with that. We have Qiime in our server through biolinux. To the best of my knowledge no one has changed or reinstalled BIOM. Should I delete an re-install?


Thank you!


Maria

Daniel McDonald

unread,
Nov 28, 2016, 6:26:07 PM11/28/16
to Qiime 1 Forum
Hi Maria,

Yes. My guess is that someone manually modified a library file as that error is very odd given a brief google search.

Best,
Daniel
Reply all
Reply to author
Forward
0 new messages