run-time vs. compile time import paths.

[Chris Barker]

Hi folks,

We've set up a package structure that puts all our Cython source in
one place, but then installs the actual built extensions somewhere
else -- it seems cleaner that way to us.

However, we now have a problem:

when we cimport something, Cython needs the *.pyd file at compile
time, but then it also puts a run-tiem import inot teh code
(presumable to import the module compiled from the corresponding *.pyx
file).

But in our case, the import path is different at run time and compile
time, i.e.:

cythonizing time:

cimport the_module

run time

from package_name.sub_package import the_module

I thought relative imports might help, but apparently not. All the
compiled modules are in the same place, so a .the_module relative
import could theoretically work.

Is there a solution to this, or should we simply follow what is
presumably the convention and build/install our compiled modules in
the same place?

-Thanks,
-Chris


--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris....@noaa.gov

[Robert Bradshaw]

On Wed, Oct 3, 2012 at 10:58 AM, Chris Barker <chris....@noaa.gov> wrote:
> Hi folks,
>
> We've set up a package structure that puts all our Cython source in
> one place, but then installs the actual built extensions somewhere
> else -- it seems cleaner that way to us.
>
> However, we now have a problem:
>
> when we cimport something, Cython needs the *.pyd file at compile
> time, but then it also puts a run-tiem import inot teh code
> (presumable to import the module compiled from the corresponding *.pyx
> file).
>
> But in our case, the import path is different at run time and compile
> time, i.e.:
>
> cythonizing time:
>
> cimport the_module
>
> run time
>
> from package_name.sub_package import the_module
>
> I thought relative imports might help, but apparently not. All the
> compiled modules are in the same place, so a .the_module relative
> import could theoretically work.
>
> Is there a solution to this, or should we simply follow what is
> presumably the convention and build/install our compiled modules in
> the same place?

Supporting relative cimports might work, and there was a recent thread
on that issue, but in the meantime compiling the modules with the same
absolute paths as they're going to be used is your best bet.
(Personally, I actually find it cleaner that way, but then again lots
of my code is a mixture of Python and Cython files that call each
other so putting stuff in a separate place just because it's compiled
would be a pain.)

- Robert

[Chris Barker]

On Wed, Oct 3, 2012 at 11:16 AM, Robert Bradshaw <robe...@gmail.com> wrote:
> Supporting relative cimports might work, and there was a recent thread
> on that issue, but in the meantime compiling the modules with the same
> absolute paths as they're going to be used is your best bet.

Fair enough.

> (Personally, I actually find it cleaner that way, but then again lots
> of my code is a mixture of Python and Cython files that call each
> other so putting stuff in a separate place just because it's compiled
> would be a pain.)

Similar for us, though it ends up with a LOT of files all in one
place, so this seemed cleaner. But it's not working, so it's hard to
call that clean!

[Robert Bradshaw]

On Wed, Oct 3, 2012 at 11:38 AM, Chris Barker <chris....@noaa.gov> wrote:
> On Wed, Oct 3, 2012 at 11:16 AM, Robert Bradshaw <robe...@gmail.com> wrote:
>> Supporting relative cimports might work, and there was a recent thread
>> on that issue, but in the meantime compiling the modules with the same
>> absolute paths as they're going to be used is your best bet.
>
> Fair enough.
>
>
>> (Personally, I actually find it cleaner that way, but then again lots
>> of my code is a mixture of Python and Cython files that call each
>> other so putting stuff in a separate place just because it's compiled
>> would be a pain.)
>
> Similar for us, though it ends up with a LOT of files all in one
> place, so this seemed cleaner. But it's not working, so it's hard to
> call that clean!

Perhaps that suggests re-organize the packaging of your modules rather
than dividing things into cython/non-cython (which is at most a
two-fold reduction in the number of files). I could be
miss-understanding your layout completely though (and things may be
different if you have a whole bunch of supporting raw c files), but
the typical setup is that .pyx files in your project are just like .py
files except that they happen to be compiled.

- Robert

[Chris Barker]

On Wed, Oct 3, 2012 at 11:50 AM, Robert Bradshaw <robe...@gmail.com> wrote:
>> Similar for us, though it ends up with a LOT of files all in one
>> place, so this seemed cleaner. But it's not working, so it's hard to
>> call that clean!
>
> Perhaps that suggests re-organize the packaging of your modules rather
> than dividing things into cython/non-cython (which is at most a
> two-fold reduction in the number of files).

Yes, we may want to sub-divide the package more, that does make sense.

> I could be
> miss-understanding your layout completely though (and things may be
> different if you have a whole bunch of supporting raw c files),

well, yes, but we've got those somewhere else anyway.

> but
> the typical setup is that .pyx files in your project are just like .py
> files except that they happen to be compiled.

True -- it probably is beter to think of it that way. We do end up with:

*.py
*.pxi
*.pxd
*.pxy
*.so (at least in develop or in-place mode)

for each module -- but not so bad, really.

[Robert Bradshaw]

On Wed, Oct 3, 2012 at 12:04 PM, Chris Barker <chris....@noaa.gov> wrote:
> On Wed, Oct 3, 2012 at 11:50 AM, Robert Bradshaw <robe...@gmail.com> wrote:
>>> Similar for us, though it ends up with a LOT of files all in one
>>> place, so this seemed cleaner. But it's not working, so it's hard to
>>> call that clean!
>>
>> Perhaps that suggests re-organize the packaging of your modules rather
>> than dividing things into cython/non-cython (which is at most a
>> two-fold reduction in the number of files).
>
> Yes, we may want to sub-divide the package more, that does make sense.
>
>> I could be
>> miss-understanding your layout completely though (and things may be
>> different if you have a whole bunch of supporting raw c files),
>
> well, yes, but we've got those somewhere else anyway.
>
>> but
>> the typical setup is that .pyx files in your project are just like .py
>> files except that they happen to be compiled.
>
> True -- it probably is beter to think of it that way. We do end up with:
>
> *.py
> *.pxi
> *.pxd
> *.pxy
> *.so (at least in develop or in-place mode)
>
> for each module -- but not so bad, really.

I am curious why you need a .py and .pxi for each module, on top of
the .pxy. Historical artifact?
(https://github.com/cython/cython/blob/master/Cython/Compiler/Options.py#L66
might be of interest as well.)

- Robert

[Chris Barker]

>> True -- it probably is beter to think of it that way. We do end up with:
>>
>> *.py
>> *.pxi
>> *.pxd
>> *.pxy
>> *.so (at least in develop or in-place mode)
>>
>> for each module -- but not so bad, really.
>
> I am curious why you need a .py and .pxi for each module, on top of
> the .pxy. Historical artifact?

This is mostly wrapping some C++ code:

The .py is there because I prefer to put the pure-python parts of our
wrappers in pure python -- it makes for easier debugging, etc. I
suppose once they are mature we could merge it, but I think I like it
this way.

the .pxi are for decalring various stuff in the C++ headers.

the .pxd is for declarations so that we can use the stuff we defined
in other pyx files.

But I'm still confused about pxi vs pxd -- maybe we could put it all
in .pxd files?

> (https://github.com/cython/cython/blob/master/Cython/Compiler/Options.py#L66
> might be of interest as well.)

ANd this indicates that we could do away with the extra declarations
in the pxd files (for our cdef classes). IN fact, that's a bit ugly
now, putting some info pxd and some in pxy -- so I"ll look into that.

Since you've taken an interest -- care to sit down an take a look at
this with us? I"ll buy the beer!

Also -- once I've got this worked out, we'll write a up a wiki page --
it has been a bit hard to figure out how to structure all this.

[Robert Bradshaw]

On Wed, Oct 3, 2012 at 2:06 PM, Chris Barker <chris....@noaa.gov> wrote:
>>> True -- it probably is beter to think of it that way. We do end up with:
>>>
>>> *.py
>>> *.pxi
>>> *.pxd
>>> *.pxy
>>> *.so (at least in develop or in-place mode)
>>>
>>> for each module -- but not so bad, really.
>>
>> I am curious why you need a .py and .pxi for each module, on top of
>> the .pxy. Historical artifact?
>
> This is mostly wrapping some C++ code:
>
> The .py is there because I prefer to put the pure-python parts of our
> wrappers in pure python -- it makes for easier debugging, etc. I
> suppose once they are mature we could merge it, but I think I like it
> this way.

Sure, that's fine rational (though ideally the incentive to do this
will get smaller and smaller).

> the .pxi are for decalring various stuff in the C++ headers.
>
> the .pxd is for declarations so that we can use the stuff we defined
> in other pyx files.
>
> But I'm still confused about pxi vs pxd -- maybe we could put it all
> in .pxd files?

Yep, it can likely all go into your pxd files.

>> (https://github.com/cython/cython/blob/master/Cython/Compiler/Options.py#L66
>> might be of interest as well.)
>
> ANd this indicates that we could do away with the extra declarations
> in the pxd files (for our cdef classes). IN fact, that's a bit ugly
> now, putting some info pxd and some in pxy -- so I"ll look into that.

That's still somewhat of an experimental feature, and there are pros
and cons of having separate declaration files, but sometimes it's nice
to not have to split things up.

> Since you've taken an interest -- care to sit down an take a look at
> this with us? I"ll buy the beer!

Sure. Don't drink, but I'm sure we could find a reasonable substitute :).

> Also -- once I've got this worked out, we'll write a up a wiki page --
> it has been a bit hard to figure out how to structure all this.

Yeah, understandable. A good writeup would be helpful to a lot of
folks I think. IIRC, you've been using Cython/Pyrex long enough that
what makes sense may have shifted since you started too...

- Robert

[Chris Barker]

On Wed, Oct 3, 2012 at 2:14 PM, Robert Bradshaw <robe...@gmail.com> wrote:

>> But I'm still confused about pxi vs pxd -- maybe we could put it all
>> in .pxd files?
>
> Yep, it can likely all go into your pxd files.

OK -- we'll give that a try.

>>> (https://github.com/cython/cython/blob/master/Cython/Compiler/Options.py#L66
>>> might be of interest as well.)
>>
>> ANd this indicates that we could do away with the extra declarations
>> in the pxd files (for our cdef classes). IN fact, that's a bit ugly
>> now, putting some info pxd and some in pxy -- so I"ll look into that.
>
> That's still somewhat of an experimental feature, and there are pros
> and cons of having separate declaration files, but sometimes it's nice
> to not have to split things up.

I like the idea of keeping it together, we'll try that.

>> Since you've taken an interest -- care to sit down an take a look at
>> this with us? I"ll buy the beer!
>
> Sure. Don't drink, but I'm sure we could find a reasonable substitute :).

Food, prune juice, coffee, whatever would be great.

>> Also -- once I've got this worked out, we'll write a up a wiki page --
>> it has been a bit hard to figure out how to structure all this.
>
> Yeah, understandable. A good writeup would be helpful to a lot of
> folks I think. IIRC, you've been using Cython/Pyrex long enough that
> what makes sense may have shifted since you started too...

That's part of it, yes. Cython has made some great strides forward.
But most of the work is being done by someone new to Cython anyway --
but a lot of the examples on the net are old, too, so ti's a bit hard
to find the latest and greatest way to do things.

[Chris Barker]

On Wed, Oct 3, 2012 at 2:14 PM, Robert Bradshaw <robe...@gmail.com> wrote:

>>> (https://github.com/cython/cython/blob/master/Cython/Compiler/Options.py#L66
>>> might be of interest as well.)

OK -- that shows the option:

# Allows cimporting from a pyx file without a pxd file.
cimport_from_pyx = False

Which looks like a great idea. However:

1) how do you change that option -- treating it like compiler
directive, either with:

#cython: cimport_from_pyx=False

or passing it in on the comamnd line:

cython --cplus -X cimport_from_pyx=True multi_rectangle.pyx


did not work.

So I changed it in the Options.py file directly. But I'd rather not
make that a global change!

After that, Cython now gets psat the cipmport statement, no longer
with the error tha the *.pxd can not be found. But it doesn't seem to
work either.

I borrowed an example from the Wiki, and am adding to it, so in
rectangle.pyx, I have:

cdef class Rectangle:
cdef c_Rectangle *thisptr # hold a C++ instance which we're wrapping
def __cinit__(self, int x0, int y0, int x1, int y1):
self.thisptr = new_Rectangle(x0, y0, x1, y1)
....
(and so on)


Then in multi_rectangle.pyx, I put:


from rectangle cimport Rectangle
...
def addRect(self, Rectangle rect):
"""
add a rectangle object -- this will only take a rectangle object
"""

self.rectangles.append(rect)
...

But I get:


def addRect(self, Rectangle rect):
^
------------------------------------------------------------

multi_rectangle.pyx:24:22: 'Rectangle' is not a type identifier


so something isn't working.

I've enclosed the whole package if anyone wants to take a look.

[Robert Bradshaw]

On Wed, Oct 3, 2012 at 5:27 PM, Chris Barker <chris....@noaa.gov> wrote:
> On Wed, Oct 3, 2012 at 2:14 PM, Robert Bradshaw <robe...@gmail.com> wrote:
>
>>>> (https://github.com/cython/cython/blob/master/Cython/Compiler/Options.py#L66
>>>> might be of interest as well.)
>
> OK -- that shows the option:
>
> # Allows cimporting from a pyx file without a pxd file.
> cimport_from_pyx = False
>
> Which looks like a great idea. However:
>
> 1) how do you change that option -- treating it like compiler
> directive, either with:
>
> #cython: cimport_from_pyx=False
>
> or passing it in on the comamnd line:
>
> cython --cplus -X cimport_from_pyx=True multi_rectangle.pyx
>
>
> did not work.
>
> So I changed it in the Options.py file directly. But I'd rather not
> make that a global change!
>
> After that, Cython now gets psat the cipmport statement, no longer
> with the error tha the *.pxd can not be found. But it doesn't seem to
> work either.

That's why it's not easy to turn on yet :-P. You can set it manually
in a setup.py by importing Cython.Compiler.Options, but it's not
something I'd worry about at this point. Thanks for the example, I'll
take a look.