Strainplan sample2markers.py error - ooSubprocessException: Failed when executing the command: dump_file.py --input_file sams/sampleE.sam.bz2

[aman]

Hi, I'm getting an error using StrainPhlan, when I run sample2markers.py. [This is a repost since the original had poor formatting]

I have looked at other messages regarding this error and the usual advice is to ensure that bcftools 0.1.19 is running. I have done that, including by pointing directly to bcftools using the bcftools_exe flag. Yet I still get the same error. Any advice on troubleshooting this would be greatly appreciated

Command: 

python2.7 ../strainphlan_src/sample2markers.py --ifn_samples sams/sampleE.sam.bz2 --input_type sam --output_dir consensus_markers --nprocs 28 --bcftools_exe ~/bin/bcftools &> consensus_markers/logE.txt

Error: 
cat consensus_markers/logE.txt 
Traceback (most recent call last):
  File "/home/ec2-user/metaphlan2/strainphlan_src/ooSubprocess.py", line 244, in wrapper
    return f(*args, **kwargs)
  File "../strainphlan_src/sample2markers.py", line 401, in run_sample
    bcftools_exe=args['bcftools_exe'])
  File "../strainphlan_src/sample2markers.py", line 252, in sam2markers
    stderr=error_pipe)
  File "/home/ec2-user/metaphlan2/strainphlan_src/ooSubprocess.py", line 181, in chain
    %(' | '.join(self.chain_cmds), return_code))
ooSubprocessException: Failed when executing the command: dump_file.py --input_file sams/sampleE.sam.bz2
return code: 1

Versions:

$ samtools
Program: samtools (Tools for alignments in the SAM format)
Version: 0.1.19-44428cd
$ ~/bin/bcftools 
Program: bcftools (Tools for data in the VCF/BCF formats)
Version: 0.1.19-44428cd

I also tried separately installing samtools-0.1.19 and using the bcftools therein (and pointing to it using --bcftools_exe) but there was no change. This is the second version of bcftools I tried:

$ temp/samtools-0.1.19/bcftools/bcftools 
Program: bcftools (Tools for data in the VCF/BCF formats)
Version: 0.1.19-44428cd

Linux OS:

$ cat /etc/os-release
NAME="Amazon Linux AMI"
VERSION="2018.03"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2018.03"
PRETTY_NAME="Amazon Linux AMI 2018.03"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2018.03:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"

[Aitor Blanco-Miguez]

Hi Aman, thank you for contacting us.
Can you report also the version of MetaPhlAn2 are you executing?

Best,

Aitor

[aman]

Hi Aitor, 

I'm running metaphlan 2.7.7 (installed via miniconda). Looking at another post about which database to download, it seems my issue is that my metaphlan version is too old to use databases v294 and v295? 

Thanks,

Aman

[Aitor Blanco-Miguez]

Hi Aman, 
The latest databases (v294 and v295) are compatible only with MetaPhlAn2 v2.9. You can get it from the 2.9 branch in the Bitbucket repository or from Bioconda.
I hope this solve your issue.

Best, 
Aitor

[Christophe Lemetre]

Hi Aitor,

Many thanks for the input. I just installed Metaphlan2 from conda, but it only installed the 2.7 version... How do I force the install of version 2.9?

Many thanks

Christophe

[Valentina Galata]

Check available versions online (https://anaconda.org/bioconda/metaphlan2/files) and specify them when calling conda. E.g., the latest version is 2.9.5:

conda install -c bioconda metaphlan2=2.9.5

[Christophe Lemetre]

So it seems that something is going on at the first execution of sample2markers.py. I installed the 2.9.5 version with anaconda3. bcftools is present in the bin directory:

$ ll /Data-2/anaconda3/envs/metag/bin/bcft*
total 131396
-rwxrwxr-x  1 ubuntu ubuntu   607112 Dec  3 17:07 bcftools*
-rw-rw-r--  1 ubuntu ubuntu   240934 Dec  3 16:57 bcftools_0.1.19

I do find bcftools (using which):

(metag) ubuntu@ip-172-31-1-194$ which bcftools
/Data-2/anaconda3/envs/metag/bin/bcftools

However as soon as I execute sample2markers, the exe is no longer found:

(metag) ubuntu@ip-172-31-1-194$ sample2markers.py --ifn_samples 13530241_SF05.sam.bz2 --input_type sam --output_dir strainphlan_output/

Traceback (most recent call last):

  File "/Data-2/anaconda3/envs/metag/bin/ooSubprocess.py", line 246, in wrapper
    return f(*args, **kwargs)
  File "/Data-2/anaconda3/envs/metag/bin/sample2markers.py", line 414, in run_sample
    bcftools_exe=args['bcftools_exe'])
  File "/Data-2/anaconda3/envs/metag/bin/sample2markers.py", line 333, in sam2markers
    stderr=error_pipe)
  File "/Data-2/anaconda3/envs/metag/bin/ooSubprocess.py", line 105, in chain
    'in the executable path!'%prog)
ooSubprocess.ooSubprocessException: Error: cannot find the program bcftools in the executable path!

looking again for the bcftools exe after running this command, it has been replaced by a much smaller file (a symlink):

(metag) ubuntu@ip-172-31-1-194:$ ll /Data-2/anaconda3/envs/metag/bin/bcf*
lrwxrwxrwx 1 ubuntu ubuntu     48 Dec  3 17:12 /Data-2/anaconda3/envs/metag/bin/bcftools -> /Data-2/anaconda3/envs/metag/bin/bcftools_0.1.19
-rw-rw-r-- 1 ubuntu ubuntu 240934 Dec  3 16:57 /Data-2/anaconda3/envs/metag/bin/bcftools_0.1.19

Note the time stamp on the bcftools exe has changed after the execution of sample2markers. So sample2markers does remove the bcftools exe installed in the first place and then throws an error that the exe is not found...

[Aitor Blanco-Miguez]

Hi Christophe,

Could you try to specify on the command line the "--bcftools_exe" parameter?
--bcftools_exe /Data-2/anaconda3/envs/metag/bin/bcftools

Best,

Aitor

[Christophe Lemetre]

I did, but it didn't matter, the exe was gone as I showed above. Looking for bcftools before and after executing sample2markers.py, the file is changed and disappeared (see the size going from 607112 to 48 (symlink). 
I have installed bcftools separately now and call it from the option in the script.

However, I am now hitting another hicup but I created a new thread since it is a different issue in the execution of the same script.

Thank you!
Christophe