Can install pysam with 2.7.10 but not 3.4.3 (using Anaconda)

[John Blischak]

Hi,

I am having trouble installing pysam for use with Python 3.4.3. I am confused because I have no trouble doing it with Python 2.7.10. I am using the Anaconda distribution from Continuum because I do not have administrative rights on the machine.

http://continuum.io/downloads

After installing Python 2.7.10 (bash Anaconda-2.2.0-Linux-x86_64.sh), I simply run the following and it works:

conda install pysam

After installing Python 3.4.3 (bash Anaconda3-2.2.0-Linux-x86_64.sh), I receive the following error:

conda install pysam
Fetching package metadata: ....
Solving package specifications: ..
Error: Unsatisfiable package specifications.
Generating hint:
[      COMPLETE      ] |############################################################################| 100%


Hint: the following combinations of packages create a conflict with the
remaining packages:
  - python 3.4*
  - pysam

In the pysam FAQ, it recommends using the latest version of Cython, so I updated it from 0.22 to 0.22.1 (conda update cython). However I still received the same error message when trying to install pysam.

http://pysam.readthedocs.org/en/latest/faq.html#pysam-won-t-compile

Next I tried using pip instead of conda for the installation. I've attached the full error message, but the first errors reported are below:

    package init file 'htslib/__init__.py' not found (or not a regular file)
    package init file 'htslib/htslib/__init__.py' not found (or not a regular file)
    package init file 'samtools/__init__.py' not found (or not a regular file)
    package init file 'samtools/win32/__init__.py' not found (or not a regular file)

A few others have reported similar errors in the past.

https://groups.google.com/forum/#!searchin/pysam-user-group/package$20init$20file$20$27htslib$2F__init__.py$27$20not$20found$20%28or$20not$20a$20regular$20file%29/pysam-user-group/vw_ml6GE0qI/9f8wikY7apAJ
https://groups.google.com/forum/#!searchin/pysam-user-group/package$20init$20file$20$27htslib$2F__init__.py$27$20not$20found$20%28or$20not$20a$20regular$20file%29/pysam-user-group/_ST9VXJbx1w/64APnEBTq7kJ
https://groups.google.com/forum/#!topic/pysam-user-group/sCsmBRnY2-c

One solution was to install python-devel, but I don't see how this could be the problem because 1) it worked using with Python 2.7.10, so presumably the Anaconda distribution includes python-devel, and 2) I manually checked and the file include/python3.4m/Python.h is included in the download.

Another solution was to do a local install so that one has write access, but that is what I am already doing by using Anaconda.

I am running 64-bit RHSL on a lustre filesystem, but I confirmed that the same problem exists on my 64-bit Ubuntu 14.04 machine as well.

As suggested in the FAQ, I am using the latest version of Python and Cython. I'd appreciate any advice.

Thanks,

John

[Marcel Martin]

On 2015-06-23 16:57, John Blischak wrote:
> Next I tried using pip instead of conda for the installation. I've
> attached the full error message, but the first errors reported are below:
>
> package init file 'htslib/__init__.py' not found (or not a regular
> file)
> package init file 'htslib/htslib/__init__.py' not found (or not a
> regular file)
> package init file 'samtools/__init__.py' not found (or not a
> regular file)
> package init file 'samtools/win32/__init__.py' not found (or not a
> regular file)

These aren’t the actual error messages. Your real error is:
cc1: error: unrecognized command line option
"-Wno-error=declaration-after-statement"

There’s even an issue about that here, but it’s not helpful:
https://github.com/pysam-developers/pysam/issues/36

Can you switch to a more recent compiler?

Marcel

[John Blischak]

Thanks for the quick reply, Marcel.

On Tuesday, June 23, 2015 at 10:20:45 AM UTC-5, Marcel Martin wrote:

These aren’t the actual error messages. Your real error is:
cc1: error: unrecognized command line option
"-Wno-error=declaration-after-statement"

Thanks for the clarification. I was simply following the advice from the pysam FAQ: "generally the cause of the problem is among the first errors to be reported".
 

There’s even an issue about that here, but it’s not helpful:
https://github.com/pysam-developers/pysam/issues/36

Good to know I am not the only one having this issue.
 

Can you switch to a more recent compiler?

I could try, but the installation directions for gcc appear quite involved.

https://gcc.gnu.org/install/

I realize that my work machine is (frustratingly) outdated, and I should have no expectations that it should be supported (it is running gcc 4.1.2, which was released in 2006). However, my local computer is running the latest LTS version of Ubuntu. It is running gcc 4.8.4, which was released in December 2014. While there have been two subsequent releases of gcc, this is the latest version in the Ubuntu repositories (apt-get upgrade gcc).

If installation of gcc from source is a potential prerequisite to install pysam for Python 3, I think it'll be easier to just switch back to Python 2.

Thanks,

John

[Marcel Martin]

On 2015-06-23 21:00, John Blischak wrote:
> I could try, but the installation directions for gcc appear quite involved.
>
> https://gcc.gnu.org/install/

Sure, I wouldn’t expect anyone to do that just to get pysam installed.

> I realize that my work machine is (frustratingly) outdated, and I should
> have no expectations that it should be supported (it is running gcc
> 4.1.2, which was released in 2006). However, my local computer is
> running the latest LTS version of Ubuntu. It is running gcc 4.8.4, which
> was released in December 2014. While there have been two subsequent
> releases of gcc, this is the latest version in the Ubuntu repositories
> (apt-get upgrade gcc).

I think the switch that gives you trouble was introduced already a few
years back (although definitely after 2006). Perhaps the error is a
different one on your local machine? You could post the full log of that
as well.

Marcel

[John Blischak]

Thanks for your continued advice, Marcel. I've delved into this deeper
to see if I can pinpoint the problem using the scripts at the link
below.

https://gist.github.com/jdblischak/035449e1469664810536

On Tue, Jun 23, 2015 at 4:53 PM, Marcel Martin
<marcel...@scilifelab.se> wrote:
> I think the switch that gives you trouble was introduced already a few years
> back (although definitely after 2006). Perhaps the error is a different one
> on your local machine? You could post the full log of that as well.

On Ubuntu 14.04, I can install pysam using the following methods:

* virtual environment + pip using Python 3.3.6, 3.4.0, 3.4.1, 3.4.2, or 3.4.3
* conda using Anaconda-2.2.0 and Python 2.7.10
* pip using Anaconda-2.2.0 and Python 3.4.3

I cannot install pysam with conda using Anaconda-2.2.0 and Python
3.4.3. The error message is not very informative, but I've attached it
anyway (ubuntu14.04-py3.4.3-anaconda2.2.0-conda.txt).

On my work cluster, where I really need local installation options
like virtual environments to work because I do not have administrative
rights, it is a real mess. I can install pysam with conda using
Anaconda-2.2.0 and Python 2.7.10. Unfortunately I cannot get any
Python3 option to work. I tried the following:

* virtual environment + pip using Python 3.3.6 or 3.4.3
* conda using Anaconda-2.2.0 and Python 3.4.3
* pip using Anaconda-2.2.0 and Python 3.4.3

I've attached the output from these attempts in case it could be informative.

Thanks,

John

[Marcel Martin]

Hi John,

On 2015-06-25 19:08, John Blischak wrote:
[...]

> I've attached the output from these attempts in case it could be informative.

I started to write a long answer but then I realized what the problem
is: The "-Wno-error=declaration-after-statement" option is hardcoded in
pysam’s setup.py script.

Python modules are usually built with the same options that are used to
compile Python itself. It would therefore be strange that your pysam
compilation chokes on the above option. The explanation of course is
that pysam overrides it. I think you should file a bug report for this.

One solution is to install pysam by hand (download, unpack, 'python3
setup.py install' etc.) and modify the setup.py before you run it.
Search for extra_compile_args= and remove the above option from the list.

I don’t have experience with anaconda, so I cannot help with that.

Regards,
Marcel