Silent error after "Help message for read_fastx.py"

[Wesley Loo]

Hello,

I installed metaphlan2 from source (Version 2.7.6) and am attempting to run it with some of the example data using the following command:
metaphlan2.py SRS014476-Supragingival_plaque.fasta.gz --input_type fasta > test.txt

After 12 hours, there is still no output in 'SRS014476-Supragingival_plaque.fasta.gz.bowtie2out.txt' and here standard out only shows "Help message for read_fastx.py"

When I interrupt the process, here's the traceback:

^CTraceback (most recent call last):
File "/n/home09/wloo/biobakery-metaphlan2-7df8753d7e5e/metaphlan2.py", line 1564, in <module>
Traceback (most recent call last):
File "/n/home09/wloo/biobakery-metaphlan2-7df8753d7e5e/utils/read_fastx.py", line 123, in <module>
metaphlan2()
File "/n/home09/wloo/biobakery-metaphlan2-7df8753d7e5e/metaphlan2.py", line 1449, in metaphlan2
min_alignment_len=pars['min_alignment_len'], read_min_len=pars['read_min_len'])
File "/n/home09/wloo/biobakery-metaphlan2-7df8753d7e5e/metaphlan2.py", line 896, in run_bowtie2
for line in p.stdout:
KeyboardInterrupt
read_and_write_raw(f, opened=False, min_len=min_len)
File "/n/home09/wloo/biobakery-metaphlan2-7df8753d7e5e/utils/read_fastx.py", line 89, in read_and_write_raw
(humann2) [wloo@holy7c19316 metaphlan]$ read_and_write_raw_int(inf, min_len=min_len)
File "/n/home09/wloo/biobakery-metaphlan2-7df8753d7e5e/utils/read_fastx.py", line 78, in read_and_write_raw_int
SeqIO.write(record, sys.stdout, fmt)
File "/n/home09/wloo/.conda/envs/humann2/lib/python2.7/site-packages/Bio/SeqIO/__init__.py", line 491, in write
count = writer_class(fp).write_file(sequences)
File "/n/home09/wloo/.conda/envs/humann2/lib/python2.7/site-packages/Bio/SeqIO/Interfaces.py", line 214, in write_file
count = self.write_records(records)
File "/n/home09/wloo/.conda/envs/humann2/lib/python2.7/site-packages/Bio/SeqIO/Interfaces.py", line 199, in write_records
self.write_record(record)
File "/n/home09/wloo/.conda/envs/humann2/lib/python2.7/site-packages/Bio/SeqIO/FastaIO.py", line 200, in write_record
self.handle.write(">%s\n" % title)
KeyboardInterrupt

Do you have any suggestions for solving this?

Thank you!
Wesley

[Francesco Asnicar]

Hello Wesley,

I think the problem is that you're passing your metagenome in a compressed format. According to the MetaPhlAn2 readme file (https://bitbucket.org/biobakery/metaphlan2/#markdown-header-basic-usage, "Basic Usage" section), compressed formats can be used if decompressed and streamed to MetaPhlAn2. Following the examples in the above section, you can try with the following command:

 $ metaphlan2.py --input_type fasta <(zcat SRS014476-Supragingival_plaque.fasta.gz) > test.txt

We should probably add a check for this kind of situation, to generate an error in case of wrong input format and not keep the user waiting.

Many thanks,

Francesco

--
You received this message because you are subscribed to the Google Groups "MetaPhlAn-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to metaphlan-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Wesley Loo]

Hello Francesco,

Thank you for your quick response! I tried your suggested command but ran into the same problem. I also decompressed the fasta file and ran the command, but I'm still getting no output. The bowtie2out.txt file is created but nothing is written to it and stderr still only shows "Help message for read_fastx.py".

Do you have any other ideas for what might be causing this issue?

Thanks,
Wesley

[Francesco Asnicar]

Hello Wesley,

That's strange! I just tested the following command using the uncompressed input:

 $ metaphlan2.py --input_type fasta tests/SRS014476-Supragingival_plaque.fasta

 and I got the correct output (it takes about 4 minutes to finish the analysis).

If I run the following command (that should deal with decompressed files from the stream):

 $ metaphlan2.py --input_type fasta <(zcat tests/SRS014476-Supragingival_plaque.fasta.gz)

I got this error:

Traceback (most recent call last):

  File "./utils/read_fastx.py", line 123, in <module>

    read_and_write_raw(f, opened=False, min_len=min_len)

  File "./utils/read_fastx.py", line 88, in read_and_write_raw

    with fopen(fd) as inf:

  File "./utils/read_fastx.py", line 47, in fopen

    return open(fn)

FileNotFoundError: [Errno 2] No such file or directory: '/dev/fd/63'

A problem with the "read_fastx.py" script and I look into solving this soon.

The MetaPhlAn2 version I'm running is the 2.7.6. (5 March 2018), and if you just installed it from the Bitbucket repository should correspond. Can you please double check that there are no other temp files that MetaPhlAn2 creates when running and try again with the uncompressed file?

Many thanks and sorry for the trouble,

Francesco

[fw...@csu.fullerton.edu]

Hello Wesley,

I am just a new user of metaphlan2. I also pass the decompressed file as Francesco suggested:
<(zcat file1 file2)

On our server it takes a while before the bowtie2 output files increase in size. There will be output after some time. The read_fastx.py seems to be a standard output message. It took about 8hours for 4gb of data.

Perhaps wait a little longer and see? In my experience metaphlan v1 ran much faster.

Foon

[Wesley Loo]

Hi Francesco,

I did a clean reinstall and it all seems to be working now. I think perhaps I interrupted the database downloading/building at some point in my previous installation. Thanks for your help!

Best,
Wesley

[Greg P]

How can it be that the problem is the compressed input file when this command is literally the first example in the Metaphlan2 tutorial?

metaphlan2.py SRS014476-Supragingival_plaque.fasta.gz --input_type fasta > SRS014476-Supragingival_plaque_profile.txt

See

https://bitbucket.org/biobakery/biobakery/wiki/metaphlan2#rst-header-create-taxonomic-profiles

[Greg P]

For the same command (first call to metaphlan2.py in the tutorial) I am getting the same behavior (it prints the "Help message" and freezes). This is Metaphlan2 version 2.7.6.

This is the case for both a Linux machine running Python 3.6.5 and Bowtie2 2.3.2, as well as a Mac OS X machine running Python 3.5.2 and Bowtie2 2.3.1.

It still happens when I decompress the FASTA file (SRS014476-Supragingival_plaque) and use the uncompressed FASTA file, or even just the first 10 or so reads from that FASTA file.

I believe that the problem somehow involves interprocess communications with Bowtie2, which remains running at 300% CPU the whole time. If you kill Bowtie2, it exits the infinite loop.

I don't have this issue with Metaphlan2 version 2.6.0 (on Mac OS X).

[Francesco Asnicar]

Hello,

The problem is not with compressed formats per se, is when you stream them to MetaPhlAn2. The read_fastx.py utility is taking care automatically of the several input types that MetaPhlAn2 can work with, but it will be a problem if instead of an actual file the input is a stream (a file descriptor already open).

For the problem you mention in your second email, can you provide some more info? I think it will be helpful, for instance, if you can provide the actual commands you're issuing.

Many thanks,

Francesco

[Greg P]

The problem in my second email is the same problem - just trying the command with uncompressed input, with a shorter input, etc.

The command is from the "Run a Single Sample" part of the tutorial (https://bitbucket.org/biobakery/biobakery/wiki/metaphlan2). Namely:

metaphlan2.py SRS014476-Supragingival_plaque.fasta.gz --input_type fasta > SRS014476-Supragingival_plaque_profile.txt

With version 2.6.0 this runs in under 2 minutes. With 2.7.6 it gives the "Help message for read_fastx.py" and then freezes (with bowtie2-align-s running meanwhile).

The same thing happens if I run 2.7.6 on the uncompressed file:

gzcat SRS014476-Supragingival_plaque.fasta.gz > SRS014476-Supragingival_plaque.fasta

metaphlan2.py SRS014476-Supragingival_plaque.fasta --input_type fasta > SRS014476-Supragingival_plaque_profile.txt

Just to make sure it was not getting stuck on some weird read in the middle of the fasta file, I also took the first 10 reads:

head -n 20 SRS014476-Supragingival_plaque.fasta > tenreads.fasta

metaphlan2/metaphlan2.py tenreads.fasta --input_type fasta > profile.txt

Same problem.
-Greg