Can I combine files from different samples without a barcode?

[Luz Chacón Jiménez]

Hi!

I have a problem with my samples. I have metagenomic data of 24 samples, I have the original sff files, I converted the files is fna and fasta.qual files but when I try to split libraries my barcodes aren't found in my sequences.

But, I have files with the sequences, without barcodes, splitted by sample in fasta and fasta quality format. Can I use these files for doing analysis in Qiime?

What and how can I do it?

[KLR]

Hi Luz,

You say you have multiple sff files - has your sequencing data already been demultiplexed by you or your sequencing provider?

It is possible to work with already demultiplexed data (I have done it for Illumina data), but you will need to do a few work around steps first. I am fairly confident that my method is valid, but I am new to QIIME, so hopefully more experienced users will comment on whether this looks ok to them. 

1) Join paired reads (if applicable) and perform preliminary quality filtering

2) Combine individual sample files together in the one file using multiple_split_libraries_fastq.py (use convert_fastaqual_fastq.py first if necessary). Use the  with --demultiplexing_method_sampleid_by_file flag to add a unique sample ID to each read header. There seems to be a bug in the quality filtering steps of this script that can cause it to filter out reads of acceptable quality, so I suggest passing a parameters file with very loose quality filtering parameters to get around this issue ( max_bad_run_length, min_per_read_length_fraction, sequence_max_n are the ones to adjust). Just make sure that you have adequate quality control steps included elsewhere in your analysis pipeline.

This should get your data into a format that is suitable for QIIME. If you are using a mapping file in subsequent steps, you will still need a BarcodeSequence column, but you can leave the cells blank. You will also need to make sure that your sample names in the SampleID column match the file name identifiers that were added to the read headers using   --demultiplexing_method_sampleid_by_file. 

[zhiying Guo]

Hello Luz,

I think this script woks best on your data. http://qiime.org/scripts/add_qiime_labels.html. I have processed my data with it. This script is to generate a QIIME format file with demuliplexed sequence file.

Best regards,

Zhiying Guo 

[Luz Chacón Jiménez]

Thanks, it's work.

But now, I have another error when I tried to generate OTUS:

inisa@inisa-B85M-D3H:~/Datos_Metagenomica/Datos_BarCode/combined_fasta$  qiime > pick_de_novo_otus.py -i/home/inisa/Datos_Metagenomica/Datos_BarCode/combined_fasta/combined_seqs.fna -o otus -f
Traceback (most recent call last):
  File "/usr/lib/qiime/bin/pick_de_novo_otus.py", line 134, in <module>
    main()   
  File "/usr/lib/qiime/bin/pick_de_novo_otus.py", line 131, in main
    status_update_callback=status_update_callback)
  File "/home/inisa/.local/lib/python2.7/site-packages/qiime/workflow/upstream.py", line 306, in run_pick_de_novo_otus
    close_logger_on_success=close_logger_on_success)
  File "/home/inisa/.local/lib/python2.7/site-packages/qiime/workflow/util.py", line 122, in call_commands_serially
    raise WorkflowError(msg)
qiime.workflow.util.WorkflowError:

*** ERROR RAISED DURING STEP: Pick OTUs
Command run was:
 pick_otus.py -i /home/inisa/Datos_Metagenomica/Datos_BarCode/combined_fasta/combined_seqs.fna -o otus/uclust_picked_otus
Command returned exit status: 1
Stdout:

Stderr
Traceback (most recent call last):
  File "/usr/lib/qiime/bin/pick_otus.py", line 20, in <module>
    from qiime.util import get_tmp_filename
ImportError: cannot import name get_tmp_filename

What can I do?

[Colin Brislawn]

Hello Luz,

Based on that error, I wonder if there is a problem with your qiime config file. Can you run this command for me a post the result?

print_qiime_config.py -t

Thanks!
Colin

[Luz Chacón Jiménez]

System information
==================
         Platform:    linux2
   Python version:    2.7.10 (default, Oct 14 2015, 16:09:02)  [GCC 5.2.1 20151010]
Python executable:    /usr/bin/python

Dependency versions
===================
                     PyCogent version:    1.5.3
                        NumPy version:    1.10.4
                   matplotlib version:    1.5.1
                  biom-format version:    2.1.5
                         qcli version:    0.1.1
                QIIME library version:    1.9.1
                 QIIME script version:    1.8.0+dfsg-4
        PyNAST version (if installed):    1.2.2
                      Emperor version:    0.9.51
RDP Classifier version (if installed):    rdp_classifier-2.2.jar
          Java version (if installed):    1.7.0_95

QIIME config values
===================
                     blastmat_dir:    /home/inisa/qiime_software/blast-2.2.22-release/data
      pick_otus_reference_seqs_fp:    /home/inisa/.local/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
                         sc_queue:    all.q
      topiaryexplorer_project_dir:    None
     pynast_template_alignment_fp:    /home/inisa/.local/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.pynast.fasta
                  cluster_jobs_fp:    start_parallel_jobs.py
pynast_template_alignment_blastdb:    None
assign_taxonomy_reference_seqs_fp:    /home/inisa/.local/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
                     torque_queue:    friendlyq
                    jobs_to_start:    1
                       slurm_time:    None
            denoiser_min_per_core:    50
assign_taxonomy_id_to_taxonomy_fp:    /home/inisa/.local/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt
                         temp_dir:    /tmp/
                     slurm_memory:    None
                      slurm_queue:    None
                      blastall_fp:    /home/inisa/qiime_software/blast-2.2.22-release/bin/blastall
                 seconds_to_sleep:    1
F...F....F...E..FF.FF...FF.FF...F..
======================================================================
ERROR: test_denoiser_supported_version (__main__.QIIMEDependencyFull)
denoiser aligner is ready to use
----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/lib/qiime/bin/print_qiime_config.py", line 572, in test_denoiser_supported_version
    check_flowgram_ali_exe()
  File "/home/inisa/.local/lib/python2.7/site-packages/qiime/denoiser/utils.py", line 79, in check_flowgram_ali_exe
    "variable." % ali_exe)
ApplicationNotFoundError: The alignment program FlowgramAli_4frame is not accessible via the PATH environment variable.

======================================================================
FAIL: test_FastTree_supported_version (__main__.QIIMEDependencyFull)
FastTree is in path and version is supported
----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/lib/qiime/bin/print_qiime_config.py", line 364, in test_FastTree_supported_version
    % ('.'.join(map(str,acceptable_version)), version_string))
AssertionError: Unsupported FastTree version. 2.1.3 is required, but running 2.1.8.

======================================================================
FAIL: test_ampliconnoise_install (__main__.QIIMEDependencyFull)
AmpliconNoise install looks sane.
----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/lib/qiime/bin/print_qiime_config.py", line 393, in test_ampliconnoise_install
    "$PYRO_LOOKUP_FILE variable is not set to an existing filepath.")
AssertionError: $PYRO_LOOKUP_FILE variable is not set to an existing filepath.

======================================================================
FAIL: test_cdhit_supported_version (__main__.QIIMEDependencyFull)
cd-hit is in path and version is supported
----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/lib/qiime/bin/print_qiime_config.py", line 626, in test_cdhit_supported_version
    "which components of QIIME you plan to use.")
AssertionError: cd-hit not found. This may or may not be a problem depending on which components of QIIME you plan to use.

======================================================================
FAIL: test_matplotlib_suported_version (__main__.QIIMEDependencyFull)
matplotlib version is supported
----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/lib/qiime/bin/print_qiime_config.py", line 319, in test_matplotlib_suported_version
    '.'.join(map(str,max_acceptable_version)), version_string))
AssertionError: Unsupported matplotlib version. Must be >= 1.1.0 and <= 1.3.1 , but running 1.5.1.

======================================================================
FAIL: test_mothur_supported_version (__main__.QIIMEDependencyFull)
mothur is in path and version is supported
----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/lib/qiime/bin/print_qiime_config.py", line 547, in test_mothur_supported_version
    "which components of QIIME you plan to use.")
AssertionError: mothur not found. This may or may not be a problem depending on which components of QIIME you plan to use.

======================================================================
FAIL: test_numpy_suported_version (__main__.QIIMEDependencyFull)
numpy version is supported
----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/lib/qiime/bin/print_qiime_config.py", line 301, in test_numpy_suported_version
    version_string))
AssertionError: Unsupported numpy version. Must be >= 1.5.1 and <= 1.7.1 , but running 1.10.4.

======================================================================
FAIL: test_pplacer_supported_version (__main__.QIIMEDependencyFull)
pplacer is in path and version is supported
----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/lib/qiime/bin/print_qiime_config.py", line 655, in test_pplacer_supported_version
    "which components of QIIME you plan to use.")
AssertionError: pplacer not found. This may or may not be a problem depending on which components of QIIME you plan to use.

======================================================================
FAIL: test_python_exe_fp (__main__.QIIMEDependencyFull)
python_exe_fp is set to a working python env
----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/lib/qiime/bin/print_qiime_config.py", line 377, in test_python_exe_fp
    self.fail("Calling python failed. Check you python_exe_fp:%s" %python)
AssertionError: Calling python failed. Check you python_exe_fp:None

======================================================================
FAIL: test_python_supported_version (__main__.QIIMEDependencyFull)
python is in path and version is supported
----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/lib/qiime/bin/print_qiime_config.py", line 282, in test_python_supported_version
    % ('.'.join(map(str,acceptable_version)), version_string))
AssertionError: Unsupported python version. 2.7.3 is required, but running 2.7.10.

======================================================================
FAIL: test_raxmlHPC_supported_version (__main__.QIIMEDependencyFull)
raxmlHPC is in path and version is supported
----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/lib/qiime/bin/print_qiime_config.py", line 585, in test_raxmlHPC_supported_version
    "which components of QIIME you plan to use.")
AssertionError: raxmlHPC not found. This may or may not be a problem depending on which components of QIIME you plan to use.

======================================================================
FAIL: test_rtax_supported_version (__main__.QIIMEDependencyFull)
rtax is in path and version is supported
----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/lib/qiime/bin/print_qiime_config.py", line 634, in test_rtax_supported_version
    "which components of QIIME you plan to use.")
AssertionError: rtax not found. This may or may not be a problem depending on which components of QIIME you plan to use.

======================================================================
FAIL: test_uclust_supported_version (__main__.QIIMEDependencyFull)
uclust is in path and version is supported
----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/lib/qiime/bin/print_qiime_config.py", line 260, in test_uclust_supported_version
    % ('.'.join(map(str,acceptable_version)), version_string))
AssertionError: Unsupported uclust version. 1.2.22 is required, but running
UClust is not freely redistributable and is thus not included in the default QIIME package.

To use this feature, please go to:
    http://www.drive5.com/uclust/downloads1_2_22q.html

Download the 64-bit binary, then:
    sudo cp uclustq1.2.22_* /usr/local/bin/uclust
    sudo chmod a+x /usr/local/bin/uclust

.

----------------------------------------------------------------------
Ran 35 tests in 0.785s

FAILED (failures=12, errors=1)

[Colin Brislawn]

Thanks for posted that. 

I'm not totally sure what the problem is, but these lines caught my attention.

AssertionError: Unsupported FastTree version. 2.1.3 is required, but running 2.1.8.

AssertionError: Unsupported matplotlib version. Must be >= 1.1.0 and <= 1.3.1 , but running 1.5.1.

UClust is not freely redistributable and is thus not included in the default QIIME package.

                QIIME library version:    1.9.1

                 QIIME script version:    1.8.0+dfsg-4

How did you go about installing qiime?

Colin

[Luz Chacón Jiménez]

I used the Qiime website scripts. 

The test shown failures but the only fatal error was the denoising process. 

[Colin Brislawn]

Hello Luz,

The qiime website suggestions several different ways of deploying the whole pipeline. There are some new ways too, which we have been promoting on the forums.

http://qiime.org/install/install.html 

Can tell me more about your deployment method? Do you remember what commands you used?

Another good option would be to try a fresh install with Anaconda. I've used it successfully on three different computers, and it's pretty easy compared to the old way. 

Colin

[Luz Chacón Jiménez]

I couldn't deploy my samples. I had a problem with the original data. My barcodes never aligned with my samples, but my sequencer provider sent to me deployed sequences, by sample, in different files. 

I used add_qiime_labels.py and generated a new "deploy file" with fna format. But I couldn't generate an OTU table with pick_de_novo_otus.py. 

[jonsan]

Hi Luz, 

Going to echo Colin's suggestion to try reinstalling Qiime using Anaconda. With the number of failures you're seeing in the tests, it will be very difficult to track down the source of your data analysis problems.

-jon