multiple_join_paired_ends.py error message..

[Fabien Magne]

Hi, 

I want to analyse my data from mi-seq sequencing that are demultiplexed samples without barcodes.

I installed qiime inside an Ubuntu Linux virtual machine through QIIME Virtual Box.   

when I want to join my paired ends using multiple_join_paired_ends.py., I have an error message. 

 

qiime@qiime-190-virtual-box:~/Desktop/Analyse$ multiple_join_paired_ends.py -i V3_F357_N_V4_R805 -o join_paired_seq

Traceback (most recent call last):

  File "/usr/local/bin/multiple_join_paired_ends.py", line 201, in <module>

    main()

  File "/usr/local/bin/multiple_join_paired_ends.py", line 180, in main

    match_barcodes, barcode_indicator)

  File "/usr/local/lib/python2.7/dist-packages/qiime/workflow/preprocess.py", line 200, in get_pairs

    "and read2_indicator parameters as well.")

ValueError: Invalid filename found for splitting on input for file /home/qiime/Desktop/Analyse/V3_F357_N_V4_R805/V3_F357_N_V4_R805-234_AGTTGCTAGT_R2.fastq, check input read1_indicator and read2_indicator parameters as well.

can anyone help me please ?

Thank you 

Fab 

[Fabien Magne]

Hello, 

I come back...

I tried again with just two samples and my problem persist... as you can see..

I put the version of the qiime that I use. 

please help me.. 

thanks

qiime@qiime-190-virtual-box:~/Desktop/Analyse$ multiple_join_paired_ends.py -i essai -o join_paired_seq

Traceback (most recent call last):

  File "/usr/local/bin/multiple_join_paired_ends.py", line 201, in <module>

    main()

  File "/usr/local/bin/multiple_join_paired_ends.py", line 180, in main

    match_barcodes, barcode_indicator)

  File "/usr/local/lib/python2.7/dist-packages/qiime/workflow/preprocess.py", line 200, in get_pairs

    "and read2_indicator parameters as well.")

ValueError: Invalid filename found for splitting on input for file /home/qiime/Desktop/Analyse/essai/V3_F357_N_V4_R805-2_AGTGTGTCTA_R1.fastq, check input read1_indicator and read2_indicator parameters as well.

System information

==================

         Platform: linux2

   Python version: 2.7.3 (default, Dec 18 2014, 19:10:20)  [GCC 4.6.3]

Python executable: /usr/bin/python

QIIME default reference information

===================================

For details on what files are used as QIIME's default references, see here:

 https://github.com/biocore/qiime-default-reference/releases/tag/0.1.2

Dependency versions

===================

          QIIME library version: 1.9.1

           QIIME script version: 1.9.1

qiime-default-reference version: 0.1.2

                  NumPy version: 1.9.2

                  SciPy version: 0.15.1

                 pandas version: 0.16.1

             matplotlib version: 1.4.3

            biom-format version: 2.1.4

                   h5py version: 2.4.0 (HDF5 version: 1.8.4)

                   qcli version: 0.1.1

                   pyqi version: 0.3.2

             scikit-bio version: 0.2.3

                 PyNAST version: 1.2.2

                Emperor version: 0.9.51

                burrito version: 0.9.1

       burrito-fillings version: 0.1.1

              sortmerna version: SortMeRNA version 2.0, 29/11/2014

              sumaclust version: SUMACLUST Version 1.0.00

                  swarm version: Swarm 1.2.19 [May 26 2015 13:50:14]

                          gdata: Installed.

QIIME config values

===================

For definitions of these settings and to learn how to configure QIIME, see here:

 http://qiime.org/install/qiime_config.html

 http://qiime.org/tutorials/parallel_qiime.html

                     blastmat_dir: /qiime_software/blast-2.2.22-release/data

      pick_otus_reference_seqs_fp: /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

                         sc_queue: all.q

      topiaryexplorer_project_dir: None

     pynast_template_alignment_fp: /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.pynast.fasta

                  cluster_jobs_fp: start_parallel_jobs.py

pynast_template_alignment_blastdb: None

assign_taxonomy_reference_seqs_fp: /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

                     torque_queue: friendlyq

                    jobs_to_start: 1

                       slurm_time: None

            denoiser_min_per_core: 50

assign_taxonomy_id_to_taxonomy_fp: /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt

                         temp_dir: /tmp/

                     slurm_memory: None

                      slurm_queue: None

                      blastall_fp: /qiime_software/blast-2.2.22-release/bin/blastall

                 seconds_to_sleep: 1

QIIME base install test results

===============================

.........

----------------------------------------------------------------------

Ran 9 tests in 0.231s

OK

[TonyWalters]

Hello,

The default R1 indicator is _R1_ and the default R2 indicator is _R2_. Your files end with _R1.fastq, so the script isn't matching the filenames. Can you change the indicator values with the --read1_indicator and --read2_indicator parameters, e.g. with _R1 and _R2 instead of the defaults?

The script page has some details about how it tries to match up the reads: http://qiime.org/scripts/multiple_join_paired_ends.html

-Tony

[mohamed.man...@gmail.com]

Hi everyone,

I had the same msg error, I checked the script page Tony but I did not really understand what should I change ( I have already tried to change my file names) but still the same msg error!! is there something that escaped me !!

SOS,

Thank you

[TonyWalters]

Hello Mohamed,

Can you post the multiple_join_paired_ends.py command you are currently using (and the error)? What did you change the file names to?

How many total files are you dealing with in the /essai/ folder?

[mohamed.man...@gmail.com]

so the command is <<< multiple_join_paired_ends.py -i /home/qiime/Desktop/mock_data  -o home/qiime/Desktop/mock_data --read1_indicator '_forward_' --read2_indicator '_reverse_'>>>>

I have 6 files (in mock_data directory) see attached file. I am getting the same error msg like the other users which is 

<<<<<  File "/usr/local/bin/multiple_join_paired_ends.py", line 201, in <module>

    main()

  File "/usr/local/bin/multiple_join_paired_ends.py", line 180, in main

    match_barcodes, barcode_indicator)

  File "/usr/local/lib/python2.7/dist-packages/qiime/workflow/preprocess.py", line 200, in get_pairs

    "and read2_indicator parameters as well.")

ValueError: Invalid filename found for splitting on input for file /home/qiime/Desktop/mock_data/341F-785R-sample-51_2.fastq, check input read1_indicator and read2_indicator parameters as well.>>>>>>>

Than you for you intervention Tony

[TonyWalters]

Hello,

The --read_indicators are looking for a particular string in the filename. E.g., you specified --read1_indicator '_forward_' --read2_indicator '_reverse_', and the file name that gave the error is: 341F-785R-sample-51_2.fastq

There is no _forward_ or _reverse_ in the filename 341F-785R-sample-51_2.fastq. Can you try --read1_indicator '_1' --read2_indicator '_2' as parameters instead of --read1_indicator '_forward_' --read2_indicator '_reverse_' and see if that detects the filenames correctly?

The png file you posted has different filenames, where those the original ones? They end with _R1 and _R2 rather than _2 as in the error message, did you change all of the _R1 to _1 and _R2 to _2?

[mohamed.man...@gmail.com]

 oh sorry i sent you the msg error after I changed my file names but I got the same error msg when I used my original files name (the ones attached in the previous post)

<<<<< Traceback (most recent call last):

  File "/usr/local/bin/multiple_join_paired_ends.py", line 201, in <module>

    main()

  File "/usr/local/bin/multiple_join_paired_ends.py", line 180, in main

    match_barcodes, barcode_indicator)

  File "/usr/local/lib/python2.7/dist-packages/qiime/workflow/preprocess.py", line 200, in get_pairs

    "and read2_indicator parameters as well.")

ValueError: Invalid filename found for splitting on input for file /home/qiime/Desktop/mock_data/341F-785R-sample-51_R2.fastq, check input read1_indicator and read2_indicator parameters as well.>>>>>

[TonyWalters]

Okay, can you try this command then:

multiple_join_paired_ends.py -i /home/qiime/Desktop/mock_data  -o home/qiime/Desktop/join_paired_ends/ --read1_indicator '_R1' --read2_indicator '_R2'

Note that I set a different output folder from the input folder, it makes keeping track of data a bit easier, and you want to keep the intermediate files separate from the stitched final files.

Also, with only a few files, you could merge the files separately with a join_paired_ends.py command for each R1 and R2 file that you want to stitch, but I think the above command will detect the file names correctly.

[mohamed.man...@gmail.com]

Tony it is workingggggg.. i knew my mistake I was using read1_indicator '_forward_' --read2_indicator '_reverse_'   I changed this to _R1 ...... _R2 and it look like it works.....  

Thank you so much

[mohamed.man...@gmail.com]

exactly this is what I did...

Cheers!!!!!!

[Fab]

Thank you Tony for your answer. 

I changed the indicators values with _R1 and _R2 instead of the defaults so I resolved a part of my problem. 

because I have yet a message error.... 

I also tried to execute it with join_paired_ends. py, but I have a similar error message.

Thanks for your help

Fab

multiple_join_paired_ends.py -i Beta_glucan  -o joined_paired_seq --read1_indicator "_R1" --read2_indicator "_R2"

Traceback (most recent call last):

  File "/macqiime/anaconda/bin/multiple_join_paired_ends.py", line 201, in <module>

    main()

  File "/macqiime/anaconda/bin/multiple_join_paired_ends.py", line 198, in main

    close_logger_on_success=True)

  File "/macqiime/anaconda/lib/python2.7/site-packages/qiime/workflow/util.py", line 122, in call_commands_serially

    raise WorkflowError(msg)

qiime.workflow.util.WorkflowError: 

*** ERROR RAISED DURING STEP: join_paired_ends.py: /Users/Shared/Beta_glucan/V3_F357_N_V4_R805-168_TAACTCTGCT_R1.fastq

Command run was:

 join_paired_ends.py  -f /Users/Shared/Beta_glucan/V3_F357_N_V4_R805-168_TAACTCTGCT_R1.fastq -r /Users/Shared/Beta_glucan/V3_F357_N_V4_R805-168_TAACTCTGCT_R2.fastq -o /Users/Shared/joined_paired_seq/V3_F357_N_V4_R805-168_TAACTCTGCT_R1 

Command returned exit status: 1

Stdout:

Stderr

Traceback (most recent call last):

  File "/macqiime/anaconda/bin/join_paired_ends.py", line 178, in <module>

    main()

  File "/macqiime/anaconda/bin/join_paired_ends.py", line 155, in main

    working_dir=output_dir)

  File "/macqiime/anaconda/lib/python2.7/site-packages/bfillings/fastq_join.py", line 207, in join_paired_end_reads_fastqjoin

    result = fastq_join_app(infile_paths)

  File "/macqiime/anaconda/lib/python2.7/site-packages/burrito/util.py", line 303, in __call__

    result_paths)

  File "/macqiime/anaconda/lib/python2.7/site-packages/burrito/util.py", line 325, in _handle_app_result_build_failure

    raise ApplicationError("Error constructing CommandLineAppResult.")

burrito.util.ApplicationError: Error constructing CommandLineAppResult.

[TonyWalters]

Hmm, well that error message isn't that informative. It may be worth running the fastq-join (the default stitching software for join_paired_ends.py) command directly to see if we get a more clear idea of what is going on.

This command should work:

fastq-join /Users/Shared/Beta_glucan/V3_F357_N_V4_R805-168_TAACTCTGCT_R1.fastq /Users/Shared/Beta_glucan/V3_F357_N_V4_R805-168_TAACTCTGCT_R2.fastq -o /Users/Shared/joined_paired_seq/V3_F357_N_V4_R805-168_TAACTCTGCT_R1/fj_V3_F357_N_V4_R805-168_TAACTCTGCT

[Fab]

Hi Tony, 

When I do it I have this... permission denied..

MacQIIME Air-de-MAGNE:Shared $ fastq-join /Users/Shared/Beta_glucan/V3_F357_N_V4_R805-168_TAACTCTGCT_R1.fastq /Users/Shared/Beta_glucan/V3_F357_N_V4_R805-168_TAACTCTGCT_R2.fastq -o /Users/Shared/joined_paired_seq/V3_F357_N_V4_R805-168_TAACTCTGCT_R1/fj_V3_F357_N_V4_R805-168_TAACTCTGCT

Error opening file '/Users/Shared/Beta_glucan/V3_F357_N_V4_R805-168_TAACTCTGCT_R1.fastq': Permission denied 

[TonyWalters]

Can you ask whoever manages this system (cluster?) to give you read/write access to /Users/Shared/ (or at least read access, and you'll have to write, the -o parameter, to a folder that you have write access to)?

[Fab]

I am the cluster....
How can I do ?

[TonyWalters]

Can you clarify what you mean? Are you running this on a cluster, or on your local system? In either case, someone with admin permissions will need to use a chmod command to give you access to read those files. See this page for more details on chmod: https://www.esrl.noaa.gov/gmd/dv/hats/cats/stations/qnxman/chmod.html

[Fab]

Thank you very much Tony, 

I understood my problem!!!!

I installed qiime on my mac and I have a session dedicated to qiime, so I created a shared file to share my folder with my other session. 

And I did not have the permission in the qiime session to access to my shared foolder.

So I transferred my sequences in the " my document" file that I am authorized and I retried the command multiple_join_paired_ends. py and worked perfectly... 

In my home, I have another computer but it is a PC window7 premium, I had a similar error message. I installed the qiime virtual box. I going to see if the problem can be resolved similarly....

looking the solution I feel foolish.. lol

thank you Tony

fab