best practice to avoid recompiling of extension. Need help

[Sarvi Shanmugham]

Can't figure out how to prevent recompilation of code after building and installation

I have a development environment where I build and install libraries under ./DebugLNX a local directory

I also build and install the CFFI extension wrapper for tue library under ./DebugLNX/py_install

my cffi wrapper uses verify as follows

compat = ffi.verify('''

#include <compat.h>

                 ''', ext_package='sparse', tag='compat', tmpdir='__pycache__',

                 include_dirs = ['.',],

                 library_dirs = ['DebugLNX',],

                 extra_compile_args=[],

                 extra_link_args=[],

                 libraries=['sparse',])

1. With the I can only invoke/import this from the root directory where DebugLNX as well as all the header files exist. 

If I try to execute it from a sub directory say ./test/ It tries to recompile the library and complains it can't find "compat.h"

2.If I try to set include_dirs=[os.path.dirname(__file__) it tries to rebuild after installation into DebugLNX/py_install and it can't find the header files there

What is the best practice for doing this. to prevent it from recompiling after build and installation?

Sarvi

[Sarvi Shanmugham]

Also using setuptools instead of distuils causes an import error. Everything else remaing the same.

The setuptools version is pretty uptodate.

bash-3.00$ LD_LIBRARY_PATH=/ws/sarvi-sjc/mytools/lib:./DebugLNX: PYTHONPATH=./DebugLNX/py_install/lib/python2.7/site-packages: /ws/sarvi-sjc/mytools/bin/python

Python 2.7.2 (default, Aug 31 2011, 11:10:42)

[GCC 3.4.6 20060404 (Red Hat 3.4.6-11)] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> import setuptools

>>> setuptools.__version__

'1.4.2'

The error I am getting is

bash-3.00$ make

LD_LIBRARY_PATH=/ws/sarvi-sjc/mytools/lib:./DebugLNX: PYTHONPATH=./DebugLNX/py_install/lib/python2.7/site-packages: /ws/sarvi-sjc/mytools/bin/python setup.py install --prefix=./DebugLNX/py_install

Traceback (most recent call last):

  File "setup.py", line 68, in <module>

    token.ffi.verifier.get_extension(),

  File "/ws/sarvi-sjc/mytools/lib/python2.7/distutils/core.py", line 112, in setup

    _setup_distribution = dist = klass(attrs)

  File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 243, in __init__

  File "/ws/sarvi-sjc/mytools/lib/python2.7/distutils/dist.py", line 287, in __init__

    self.finalize_options()

  File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 275, in finalize_options

  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 2120, in require

  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 2323, in requires

  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 2312, in _dep_map

  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1167, in invalid_marker

  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1185, in evaluate_marker

ImportError: cannot import name NAME

make-3.79.1-p7: *** [pysparse] Error 1

bash-3.00$

The only difference is that I changed from distutils to setuptools

#from distutils.core import setup

from setuptools import setup

Sarvi

[Armin Rigo]

Hi Sarvi,

On Wed, Dec 18, 2013 at 8:23 AM, Sarvi Shanmugham <sarv...@gmail.com> wrote:
>> 1. With the I can only invoke/import this from the root directory where
>> DebugLNX as well as all the header files exist.
>> If I try to execute it from a sub directory say ./test/ It tries to
>> recompile the library and complains it can't find "compat.h"
>>
>> 2.If I try to set include_dirs=[os.path.dirname(__file__) it tries to
>> rebuild after installation into DebugLNX/py_install and it can't find the
>> header files there
>>
>> What is the best practice for doing this. to prevent it from recompiling
>> after build and installation?

I fear I don't see an obvious solution. These issues should be gone
with the cffi 1.0 refactoring I described, but for now you need to
hack. For example you can put a symlink "DebugLNX" in the test
directory, pointing to "../DebugLNX" ...

> Also using setuptools instead of distuils causes an import error.

The traceback you pasted doesn't contain any "cffi". It crashes
during the call to setup(). Are you sure it's related to cffi at all?
Isn't it trying to check a dependency by importing the module called
"NAME" and failing? Maybe you have left a bogus line in your
setup.py, like: install_requires=["NAME"]?


A bientôt,

Armin.

[Sarvi Shanmugham]

Like I mentioned in the previous message. 

The only difference between when it is working and when it threw the traceback is the 

#from distutils.core import setup

from setuptools import setup

I moved from using the setup from distutils.core to setuptools. 

My setup.py works fine with the setup() from distutils.core and throws a traceback for setup() from setuptools.

Also the traceback above shows token.ffi.verifier.getextension() is part of the traceback.

Since everything works with setup() from distutils.core and does not with the one from setuptools I am not sure where else to look 

but some compatibility issue between cffi and setuptools.

I am looking to setuptools for convenience of inplace development using the "develop" mode, less for CFFI wrapper development, 

but for in place development of other python modules around it. I am using a workaround to develop without having to install 

changes to these pure python modules. 

But would like to get setuptools working with "develop" mode and skip the workaround.

Sarvi

[Armin Rigo]

Hi Sarvi,

On Wed, Dec 18, 2013 at 6:59 PM, Sarvi Shanmugham <sarv...@gmail.com> wrote:
> Also the traceback above shows token.ffi.verifier.getextension() is part of
> the traceback.

No, that's slightly confusing, but if you look more carefully you'll
see that it is currently in the call to setup(). The call to
getextension() was done already. The line with "getextension()" just
happens to be the last line of the multiline call to setup().

> Since everything works with setup() from distutils.core and does not with
> the one from setuptools I am not sure where else to look
> but some compatibility issue between cffi and setuptools.

Distutils ignores "install_requires" as far as I know, so if you have
a line 'install_requires="yaddayadda"', it will work fine with
distutils and crash with setuptools because there is no module called
"yaddayadda".

I'm not saying that I know for sure that it's not the fault of cffi,
but I'm saying that you should consider the possibility that it is
not, and debug it appropriately. I cannot tell you more than I did in
the previous paragraph.


A bientôt,

Armin.