multiple_join_paired_ends.py error message..

184 views
Skip to first unread message

Fabien Magne

unread,
May 9, 2017, 9:27:42 AM5/9/17
to Qiime 1 Forum
Hi, 

I want to analyse my data from mi-seq sequencing that are demultiplexed samples without barcodes.
I installed qiime inside an Ubuntu Linux virtual machine through QIIME Virtual Box.   

when I want to join my paired ends using multiple_join_paired_ends.py., I have an error message. 
 
qiime@qiime-190-virtual-box:~/Desktop/Analyse$ multiple_join_paired_ends.py -i V3_F357_N_V4_R805 -o join_paired_seq
Traceback (most recent call last):
  File "/usr/local/bin/multiple_join_paired_ends.py", line 201, in <module>
    main()
  File "/usr/local/bin/multiple_join_paired_ends.py", line 180, in main
    match_barcodes, barcode_indicator)
  File "/usr/local/lib/python2.7/dist-packages/qiime/workflow/preprocess.py", line 200, in get_pairs
    "and read2_indicator parameters as well.")
ValueError: Invalid filename found for splitting on input for file /home/qiime/Desktop/Analyse/V3_F357_N_V4_R805/V3_F357_N_V4_R805-234_AGTTGCTAGT_R2.fastq, check input read1_indicator and read2_indicator parameters as well.

can anyone help me please ?

Thank you 

Fab 

Fabien Magne

unread,
May 9, 2017, 8:58:44 PM5/9/17
to qiime...@googlegroups.com
Hello, 

I come back...

I tried again with just two samples and my problem persist... as you can see..
I put the version of the qiime that I use. 

please help me.. 

thanks

qiime@qiime-190-virtual-box:~/Desktop/Analyse$ multiple_join_paired_ends.py -i essai -o join_paired_seq
Traceback (most recent call last):
  File "/usr/local/bin/multiple_join_paired_ends.py", line 201, in <module>
    main()
  File "/usr/local/bin/multiple_join_paired_ends.py", line 180, in main
    match_barcodes, barcode_indicator)
  File "/usr/local/lib/python2.7/dist-packages/qiime/workflow/preprocess.py", line 200, in get_pairs
    "and read2_indicator parameters as well.")
ValueError: Invalid filename found for splitting on input for file /home/qiime/Desktop/Analyse/essai/V3_F357_N_V4_R805-2_AGTGTGTCTA_R1.fastq, check input read1_indicator and read2_indicator parameters as well.


System information
==================
         Platform: linux2
   Python version: 2.7.3 (default, Dec 18 2014, 19:10:20)  [GCC 4.6.3]
Python executable: /usr/bin/python

QIIME default reference information
===================================
For details on what files are used as QIIME's default references, see here:

Dependency versions
===================
          QIIME library version: 1.9.1
           QIIME script version: 1.9.1
qiime-default-reference version: 0.1.2
                  NumPy version: 1.9.2
                  SciPy version: 0.15.1
                 pandas version: 0.16.1
             matplotlib version: 1.4.3
            biom-format version: 2.1.4
                   h5py version: 2.4.0 (HDF5 version: 1.8.4)
                   qcli version: 0.1.1
                   pyqi version: 0.3.2
             scikit-bio version: 0.2.3
                 PyNAST version: 1.2.2
                Emperor version: 0.9.51
                burrito version: 0.9.1
       burrito-fillings version: 0.1.1
              sortmerna version: SortMeRNA version 2.0, 29/11/2014
              sumaclust version: SUMACLUST Version 1.0.00
                  swarm version: Swarm 1.2.19 [May 26 2015 13:50:14]
                          gdata: Installed.

QIIME config values
===================
For definitions of these settings and to learn how to configure QIIME, see here:

                     blastmat_dir: /qiime_software/blast-2.2.22-release/data
      pick_otus_reference_seqs_fp: /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
                         sc_queue: all.q
      topiaryexplorer_project_dir: None
     pynast_template_alignment_fp: /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.pynast.fasta
                  cluster_jobs_fp: start_parallel_jobs.py
pynast_template_alignment_blastdb: None
assign_taxonomy_reference_seqs_fp: /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
                     torque_queue: friendlyq
                    jobs_to_start: 1
                       slurm_time: None
            denoiser_min_per_core: 50
assign_taxonomy_id_to_taxonomy_fp: /usr/local/lib/python2.7/dist-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt
                         temp_dir: /tmp/
                     slurm_memory: None
                      slurm_queue: None
                      blastall_fp: /qiime_software/blast-2.2.22-release/bin/blastall
                 seconds_to_sleep: 1

QIIME base install test results
===============================
.........
----------------------------------------------------------------------
Ran 9 tests in 0.231s

OK



TonyWalters

unread,
May 10, 2017, 1:56:25 AM5/10/17
to Qiime 1 Forum
Hello,

The default R1 indicator is _R1_ and the default R2 indicator is _R2_. Your files end with _R1.fastq, so the script isn't matching the filenames. Can you change the indicator values with the --read1_indicator and --read2_indicator parameters, e.g. with _R1 and _R2 instead of the defaults?

The script page has some details about how it tries to match up the reads: http://qiime.org/scripts/multiple_join_paired_ends.html

-Tony

mohamed.man...@gmail.com

unread,
May 12, 2017, 9:11:25 AM5/12/17
to Qiime 1 Forum
Hi everyone,

I had the same msg error, I checked the script page Tony but I did not really understand what should I change ( I have already tried to change my file names) but still the same msg error!! is there something that escaped me !!

SOS,

Thank you

TonyWalters

unread,
May 12, 2017, 9:16:18 AM5/12/17
to Qiime 1 Forum
Hello Mohamed,

Can you post the multiple_join_paired_ends.py command you are currently using (and the error)? What did you change the file names to?

How many total files are you dealing with in the /essai/ folder?

mohamed.man...@gmail.com

unread,
May 12, 2017, 9:34:40 AM5/12/17
to Qiime 1 Forum
so the command is <<< multiple_join_paired_ends.py -i /home/qiime/Desktop/mock_data  -o home/qiime/Desktop/mock_data --read1_indicator '_forward_' --read2_indicator '_reverse_'>>>>

I have 6 files (in mock_data directory) see attached file. I am getting the same error msg like the other users which is 
<<<<<  File "/usr/local/bin/multiple_join_paired_ends.py", line 201, in <module>
    main()
  File "/usr/local/bin/multiple_join_paired_ends.py", line 180, in main
    match_barcodes, barcode_indicator)
  File "/usr/local/lib/python2.7/dist-packages/qiime/workflow/preprocess.py", line 200, in get_pairs
    "and read2_indicator parameters as well.")
ValueError: Invalid filename found for splitting on input for file /home/qiime/Desktop/mock_data/341F-785R-sample-51_2.fastq, check input read1_indicator and read2_indicator parameters as well.>>>>>>>


Than you for you intervention Tony

Screenshot from 2017-05-12 06:29:42.png

TonyWalters

unread,
May 12, 2017, 9:41:29 AM5/12/17
to Qiime 1 Forum
Hello,

The --read_indicators are looking for a particular string in the filename. E.g., you specified --read1_indicator '_forward_' --read2_indicator '_reverse_', and the file name that gave the error is: 341F-785R-sample-51_2.fastq

There is no _forward_ or _reverse_ in the filename 341F-785R-sample-51_2.fastq. Can you try --read1_indicator '_1' --read2_indicator '_2' as parameters instead of --read1_indicator '_forward_' --read2_indicator '_reverse_' and see if that detects the filenames correctly?

The png file you posted has different filenames, where those the original ones? They end with _R1 and _R2 rather than _2 as in the error message, did you change all of the _R1 to _1 and _R2 to _2?

mohamed.man...@gmail.com

unread,
May 12, 2017, 9:53:21 AM5/12/17
to Qiime 1 Forum
 oh sorry i sent you the msg error after I changed my file names but I got the same error msg when I used my original files name (the ones attached in the previous post)

<<<<< Traceback (most recent call last):
  File "/usr/local/bin/multiple_join_paired_ends.py", line 201, in <module>
    main()
  File "/usr/local/bin/multiple_join_paired_ends.py", line 180, in main
    match_barcodes, barcode_indicator)
  File "/usr/local/lib/python2.7/dist-packages/qiime/workflow/preprocess.py", line 200, in get_pairs
    "and read2_indicator parameters as well.")
ValueError: Invalid filename found for splitting on input for file /home/qiime/Desktop/mock_data/341F-785R-sample-51_R2.fastq, check input read1_indicator and read2_indicator parameters as well.>>>>>

TonyWalters

unread,
May 12, 2017, 9:57:18 AM5/12/17
to Qiime 1 Forum
Okay, can you try this command then:
multiple_join_paired_ends.py -i /home/qiime/Desktop/mock_data  -o home/qiime/Desktop/join_paired_ends/ --read1_indicator '_R1' --read2_indicator '_R2'

Note that I set a different output folder from the input folder, it makes keeping track of data a bit easier, and you want to keep the intermediate files separate from the stitched final files.

Also, with only a few files, you could merge the files separately with a join_paired_ends.py command for each R1 and R2 file that you want to stitch, but I think the above command will detect the file names correctly.

mohamed.man...@gmail.com

unread,
May 12, 2017, 10:01:09 AM5/12/17
to Qiime 1 Forum
Tony it is workingggggg.. i knew my mistake I was using read1_indicator '_forward_' --read2_indicator '_reverse_'   I changed this to _R1 ...... _R2 and it look like it works.....  

Thank you so much

mohamed.man...@gmail.com

unread,
May 12, 2017, 10:03:05 AM5/12/17
to Qiime 1 Forum
exactly this is what I did...

Cheers!!!!!!


Fab

unread,
May 12, 2017, 10:58:44 AM5/12/17
to Qiime 1 Forum
Thank you Tony for your answer. 

I changed the indicators values with _R1 and _R2 instead of the defaults so I resolved a part of my problem. 
because I have yet a message error.... 
I also tried to execute it with join_paired_ends. py, but I have a similar error message.
Thanks for your help
Fab

multiple_join_paired_ends.py -i Beta_glucan  -o joined_paired_seq --read1_indicator "_R1" --read2_indicator "_R2"

Traceback (most recent call last):

  File "/macqiime/anaconda/bin/multiple_join_paired_ends.py", line 201, in <module>

    main()

  File "/macqiime/anaconda/bin/multiple_join_paired_ends.py", line 198, in main

    close_logger_on_success=True)

  File "/macqiime/anaconda/lib/python2.7/site-packages/qiime/workflow/util.py", line 122, in call_commands_serially

    raise WorkflowError(msg)

qiime.workflow.util.WorkflowError: 


*** ERROR RAISED DURING STEP: join_paired_ends.py: /Users/Shared/Beta_glucan/V3_F357_N_V4_R805-168_TAACTCTGCT_R1.fastq

Command run was:

 join_paired_ends.py  -f /Users/Shared/Beta_glucan/V3_F357_N_V4_R805-168_TAACTCTGCT_R1.fastq -r /Users/Shared/Beta_glucan/V3_F357_N_V4_R805-168_TAACTCTGCT_R2.fastq -o /Users/Shared/joined_paired_seq/V3_F357_N_V4_R805-168_TAACTCTGCT_R1 

Command returned exit status: 1

Stdout:


Stderr

Traceback (most recent call last):

  File "/macqiime/anaconda/bin/join_paired_ends.py", line 178, in <module>

    main()

  File "/macqiime/anaconda/bin/join_paired_ends.py", line 155, in main

    working_dir=output_dir)

  File "/macqiime/anaconda/lib/python2.7/site-packages/bfillings/fastq_join.py", line 207, in join_paired_end_reads_fastqjoin

    result = fastq_join_app(infile_paths)

  File "/macqiime/anaconda/lib/python2.7/site-packages/burrito/util.py", line 303, in __call__

    result_paths)

  File "/macqiime/anaconda/lib/python2.7/site-packages/burrito/util.py", line 325, in _handle_app_result_build_failure

    raise ApplicationError("Error constructing CommandLineAppResult.")

burrito.util.ApplicationError: Error constructing CommandLineAppResult.



log_20170512113239.txt

TonyWalters

unread,
May 12, 2017, 11:32:52 AM5/12/17
to Qiime 1 Forum
Hmm, well that error message isn't that informative. It may be worth running the fastq-join (the default stitching software for join_paired_ends.py) command directly to see if we get a more clear idea of what is going on.

This command should work:
fastq-join /Users/Shared/Beta_glucan/V3_F357_N_V4_R805-168_TAACTCTGCT_R1.fastq /Users/Shared/Beta_glucan/V3_F357_N_V4_R805-168_TAACTCTGCT_R2.fastq -o /Users/Shared/joined_paired_seq/V3_F357_N_V4_R805-168_TAACTCTGCT_R1/fj_V3_F357_N_V4_R805-168_TAACTCTGCT

Fab

unread,
May 12, 2017, 11:46:37 AM5/12/17
to Qiime 1 Forum
Hi Tony, 

When I do it I have this... permission denied..

MacQIIME Air-de-MAGNE:Shared $ fastq-join /Users/Shared/Beta_glucan/V3_F357_N_V4_R805-168_TAACTCTGCT_R1.fastq /Users/Shared/Beta_glucan/V3_F357_N_V4_R805-168_TAACTCTGCT_R2.fastq -o /Users/Shared/joined_paired_seq/V3_F357_N_V4_R805-168_TAACTCTGCT_R1/fj_V3_F357_N_V4_R805-168_TAACTCTGCT

Error opening file '/Users/Shared/Beta_glucan/V3_F357_N_V4_R805-168_TAACTCTGCT_R1.fastq': Permission denied 

TonyWalters

unread,
May 12, 2017, 11:49:32 AM5/12/17
to Qiime 1 Forum
Can you ask whoever manages this system (cluster?) to give you read/write access to /Users/Shared/ (or at least read access, and you'll have to write, the -o parameter, to a folder that you have write access to)?

Fab

unread,
May 12, 2017, 11:56:36 AM5/12/17
to Qiime 1 Forum
I am the cluster....
How can I do ?

TonyWalters

unread,
May 12, 2017, 12:11:08 PM5/12/17
to Qiime 1 Forum
Can you clarify what you mean? Are you running this on a cluster, or on your local system? In either case, someone with admin permissions will need to use a chmod command to give you access to read those files. See this page for more details on chmod: https://www.esrl.noaa.gov/gmd/dv/hats/cats/stations/qnxman/chmod.html

Fab

unread,
May 12, 2017, 1:30:06 PM5/12/17
to Qiime 1 Forum
Thank you very much Tony, 
I understood my problem!!!!

I installed qiime on my mac and I have a session dedicated to qiime, so I created a shared file to share my folder with my other session. 
And I did not have the permission in the qiime session to access to my shared foolder.
So I transferred my sequences in the " my document" file that I am authorized and I retried the command multiple_join_paired_ends. py and worked perfectly... 

In my home, I have another computer but it is a PC window7 premium, I had a similar error message. I installed the qiime virtual box. I going to see if the problem can be resolved similarly....

looking the solution I feel foolish.. lol

thank you Tony

fab
Reply all
Reply to author
Forward
0 new messages