split_library and denoise_wrapper for non-barcoded and primer trimmed sample?

[Soonio]

I am having trouble running split_library.py on non-barcoded data (non-
multiplexed) with primer already trimmed.
I tried creating and using a mapping file with empty fields and -b 0
option but with no avail.
A mapping file is also needed during a denoising step and I cannot
proceed with these without a mapping file it seems.

Essentially, I'm running split_library.py for several quality
filtering steps as well as filtering names etc.
and I am wondering if there are other ways to do this?

Many thanks in advance!

Soonio

[Tony Walters]

Hello Soonio,

Which version of QIIME are you running?  Are your sequences all together in one file, or already separated into individual files?

-Tony

[Soonio]

The sequences are already separated into individual files.
And I'm running QIIME 1.4.

Soonio

[Tony Walters]

Hello Soonio,

I would try following the process listed at the bottom of this thread:
http://groups.google.com/group/qiime-forum/browse_thread/thread/be7888a4d382fbca/ad5c73cda8f72ce6?lnk=gst&q=split+library+no+barcode#ad5c73cda8f72ce6 

As you are running QIIME 1.4.0, you probably need to replace some files.

In any case, I've attached the newer versions of the files (/qiime/ and /tests/ files, to be replaced in your /qiime/ library and /tests folders respectively).  The exact location can depend upon how QIIME was installed.

If you are using macqiime there is a different approach to replacing files, which has instructions here:  http://www.wernerlab.org/software/macqiime/fix-split_libraries-py-in-macqiime-1-4-0

-Tony

[Soonio]

Thank you. I will have a go.

Just out of curiosity, is this problem fixed in Qiime 1.5.0? because
if it is,

then I think I will first prioritise my time to install 1.5.0.

Soonio

On May 16, 6:14 pm, Tony Walters <william.a.walt...@gmail.com> wrote:
> Hello Soonio,
>

> I would try following the process listed at the bottom of this thread:http://groups.google.com/group/qiime-forum/browse_thread/thread/be788...

>
> As you are running QIIME 1.4.0, you probably need to replace some files.
>
> In any case, I've attached the newer versions of the files (/qiime/ and
> /tests/ files, to be replaced in your /qiime/ library and /tests folders
> respectively).  The exact location can depend upon how QIIME was installed.
>
> If you are using macqiime there is a different approach to replacing files,

> which has instructions here:http://www.wernerlab.org/software/macqiime/fix-split_libraries-py-in-...

>
> -Tony
>
>
>
>
>
>
>
> On Wed, May 16, 2012 at 11:02 AM, Soonio <soo...@gmail.com> wrote:
> > The sequences are already separated into individual files.
> > And I'm running QIIME 1.4.
>
> > Soonio
>
> > On May 16, 5:55 pm, Tony Walters <william.a.walt...@gmail.com> wrote:
> > > Hello Soonio,
>
> > > Which version of QIIME are you running?  Are your sequences all together
> > in
> > > one file, or already separated into individual files?
>
> > > -Tony
>
> > > On Wed, May 16, 2012 at 10:14 AM, Soonio <soo...@gmail.com> wrote:
> > > > I am having trouble running split_library.py on non-barcoded data (non-
> > > > multiplexed) with primer already trimmed.
> > > > I tried creating and using a mapping file with empty fields and -b 0
> > > > option but with no avail.
> > > > A mapping file is also needed during a denoising step and I cannot
> > > > proceed with these without a mapping file it seems.
>
> > > > Essentially, I'm running split_library.py for several quality
> > > > filtering steps as well as filtering names etc.
> > > > and I am wondering if there are other ways to do this?
>
> > > > Many thanks in advance!
>
> > > > Soonio
>
>
>

>  files_to_replace.zip
> 51KViewDownload

[Tony Walters]

Hello again Soonio,

Yes, those issues are resolved in 1.5.0, so you could avoid the manual replacements of the files if you went that route.

Best regards,

Tony Walters

[Soonio]

Hi,

I used macqiime 1.5.0 and followed your instructions and still
couldn't get it to work.

First of all, as my sequences don't have a barcode AND primer (they
have all been removed),
the map file needs to have two empty fields:

#SampleID BarcodeSequence LinkerPrimerSequence Description
Sample001
no_description


Actually, check_id_map.py with the "-b" option is happy with just no
barcodes (I tried it by putting a dummy primer sequence),
but not at all happy with both barcodes AND LinkerPrimerSequence field
missing.

split_libraries.py still does not work with the map file with no
barcodes (even though check_id_map was happy with it) even with "-b 0"
option.
And, split_libraries.py does not work with the map file with both no
barcodes AND primer missing.
It just gives an error: "ValueError: Problems were found with mapping
file, please run check_id_map.py to identify problems."

(BTW, the command I used: split_libraries.py -f S001.fna -q S001.qual -
o S001 -m map.txt -b 0)

Properly stuck.....

Soonio

[Tony Walters]

Hello Soonio,

If you add -p as an option to check_id_map.py and to split_libraries.py, how does this affect the error you are seeing?

-Tony

[Soonio]

Hi Tony,


check_id_map.py no longer complains with -b -p option,

but split_libraries.py still is not happy with this map file... gives
the same error: Problems were round with mapping file.

(split_libraries.py -f S001.fna -q S001.qual -o S001 -m map.txt -b 0 -
p)


I actually don't think -b 0 option is actually supported...

Soonio

[Tony Walters]

Soonio, please send me your mapping file that you are using (william....@gmail.com).

-Tony

[Soonio]

Just sent! Thank you!

[Tony Walters]

Hello again Soonio,

I was able to run check_id_map.py (-p -b) and split_libraries.py -b 0 -p with your mapping file and some test fasta and qual files.

Something else must be going on.  Can you run print_qiime_config.py and post the results?  I'm curious if there are older libraries floating around somewhere.

-Tony

On Thu, May 17, 2012 at 2:57 PM, Soonio <soo...@gmail.com> wrote:

Just sent! Thank you!

[Soonio]

Here it goes:

I'm running macqiime 1.5.0.


System information
==================
Platform: darwin
Python version: 2.7.1 (r271:86832, Dec 15 2011, 08:41:37) [GCC
4.0.1 (Apple Inc. build 5493)]
Python executable: /macqiime/bin/python

Dependency versions
===================
PyCogent version: 1.5.1
NumPy version: 1.5.1
matplotlib version: 1.1.0
QIIME library version: 1.4.0
QIIME script version: 1.4.0
PyNAST version (if installed): 1.1
RDP Classifier version (if installed): rdp_classifier-2.2.jar

QIIME config values
===================
blastmat_dir: None
topiaryexplorer_project_dir: None
pynast_template_alignment_fp: /macqiime/greengenes/
core_set_aligned.fasta.imputed
cluster_jobs_fp: /macqiime/QIIME/bin/
start_parallel_jobs.py
pynast_template_alignment_blastdb: None
assign_taxonomy_reference_seqs_fp: None
torque_queue: friendlyq
template_alignment_lanemask_fp: /macqiime/greengenes/
lanemask_in_1s_and_0s
jobs_to_start: 1
cloud_environment: False
qiime_scripts_dir: /macqiime/QIIME/bin/
denoiser_min_per_core: 50
working_dir: None
python_exe_fp: /macqiime/bin/python
temp_dir: /tmp/
blastall_fp: blastall
seconds_to_sleep: 60
assign_taxonomy_id_to_taxonomy_fp: None

[Antonio González Peña]

Hi Soonio,

You are still running 1.4.0:

QIIME library version: 1.4.0
QIIME script version: 1.4.0

Can you try reinstalling following these instructions?
http://www.wernerlab.org/software/macqiime/macqiime-installation

Cheers

--
Antonio González Peña
Research Assistant, Knight Lab
University of Colorado at Boulder
https://chem.colorado.edu/knightgroup/

[Soonio]

Thank you. I don't know why but somehow when I reinstalled macqiime
1.5.0, it didn't overwrite the older version.

I have just tested the new version and it seems to be working! A BIG
THANK YOU!

Soonio

On May 18, 2:01 pm, Antonio González Peña <antgo...@gmail.com> wrote:
> Hi Soonio,
>
> You are still running 1.4.0:
>                QIIME library version:  1.4.0
>                 QIIME script version:  1.4.0
>

> Can you try reinstalling following these instructions?http://www.wernerlab.org/software/macqiime/macqiime-installation

[Soonio]

Thank you!

Without your print_qiime_config.py help, I would have gone round and
round, not solving the problem!

I have now managed to run both check_id_map.py and split_libraries.py
with no problem.

But now I'm stuck with denoise_wrapper.py as it doesn't run without
primer in the map file..

Perhaps, there are new options in denoise_wrapper.py (1.5.0) which I
don't know about? (I certainly couldn't find anything in the
documentation regarding this)

Any advise would be muchly appreciated!

[Jose Carlos Clemente]

Soonio,

denoise_wrapper.py requires either the primer (-p) or the mapping file
including the primer (-m). As far as I know, this has not changed from
previous versions of Qiime.

Jose

[Soonio]

I'm trying with -p 0 option at the moment. It seems to be doing
something!

[Jose Carlos Clemente]

That will most probably not work, as you are telling denoiser your
primer is '0', which I don't think it's the case.

[Soonio]

Oh I see...

So now in 1.5.0, check_id_map.py and split_libraries.py
can bypass the issue with processing reads
with no barcodes and primers by putting relevant paramenters,
but denoise_wrapper.py cannot as of yet I guess...

Looks like I will have to add a dummy barcode and primer into
each read for this after all which I have been trying to avoid...

[Antonio González Peña]

One option is to use -notrim in sffinfo to keep the primers and use
those as your barcodes, not pretty but it should work.

[Jose Carlos Clemente]

Soonio,

actually you can use the lower-level script, denoiser.py, which does
not require a primer. Give it a try and lets us know how it goes.

Jose

On Fri, May 18, 2012 at 9:28 AM, Soonio <soo...@gmail.com> wrote:

[Soonio]

Thank you. I have just tried a few things.

Essentially I ran three runs

1. denoise_wrapper.py with -p 0
2. denoise_wrapper.py with -p AAAAAA (I put a dummy primer in each
read)
3. denoiser.py

And they all gave me different numbers of singletons, centroids:

1. 597 sing, 103 centroids
2. 633 sing, 110 centroids
3. 565 sing, 78 centroids

Presumably the first one is not quite correct as Jose said above, but
there is a significant difference between 2 and 3. Any advise on this?

[Jose Carlos Clemente]

Can you post the exact commands you used for 2 and 3?

Thanks,
Jose

[Soonio]

Here they are:

1. denoise_wrapper.py -i sample001.txt -f sample001.fna -o
sample001_denoised --force_overwrite -p 0

2. denoise_wrapper.py -i sample001.txt -f sample001_dummyprimer.fna -o
sample001_denoised_2 --force_overwrite -p AAAAAA

3. denoiser.py -i sample001.txt -f sample001.fna -o
sample001_denoised_3


On May 18, 5:41 pm, Jose Carlos Clemente <jose.cleme...@gmail.com>
wrote:

[Jose Carlos Clemente]

Ok I have confirmed it: you cannot use a dummy primer, as you would
also need to add that to the flowgrams file, and therefore you have to
use denoiser.py. Make sure whether your data is FLX or Ti though,
right now the commands you are passing assume FLX (and I would be
surprised if you are still using FLX)

Jose

[Soonio]

Thank you ever so much!