Accessing Python long type?

[Jeroen Demeyer]

Is there any way in Cython to access the Python "long" type?

Essentially, what I'm trying to do is


cdef extern from "longintrepr.h":
cdef long PyLong_SHIFT
ctypedef unsigned int digit
ctypedef class __builtin__.long [object PyLongObject]:
cdef digit* digits

cdef extern from "Python.h":
cdef _PyObject_NewVar(type t, Py_ssize_t size)

def foo(s):
cdef long L = _PyObject_NewVar(long, s)


The problem is that Cython always thinks that "long" means C long, while
I really want Python long. Is there any solution for this?

[Stefan Behnel]

Well - what do you need it for?

Cython is actually pretty good at optimising PyLong values, and fiddling
with them manually is quite risky as their memory layout depends on a
CPython compile time setting (digit size).

Stefan

[Sturla Molden]

On 25/02/15 15:15, Stefan Behnel wrote:

> Cython is actually pretty good at optimising PyLong values, and fiddling
> with them manually is quite risky as their memory layout depends on a
> CPython compile time setting (digit size).

It seems the question is how to cdef a PyLong, since "cdef long" will
cdef a C long. It is similar to how to cdef a PyInt or PyFloat. I must
say I have never thought about it. But if Cython can optimize operations
on these elementary Python types there must be a way to do this.

Sturla

[Stefan Behnel]

It applies this special casing internally. No need to declare them.

Stefan

[Robert Bradshaw]

It's a bit of a pain, because the parser has special code to deal with
these non-whitespace-delimited types (e.g. long long int). To get
around this you can make the declarations in a pxd file and use fully
qualified cimported names, e.g. put

cdef extern from "longintrepr.h":
cdef long PyLong_SHIFT
ctypedef unsigned int digit
ctypedef class __builtin__.long [object PyLongObject]:
cdef digit* digits

in pylong.pxd and then do

cimport pylong
cdef pylong.long foo = ...

Unfortunately, it looks like there's still other issues with dealing
with variable sized types...

[Jeroen Demeyer]

On 2015-02-25 15:15, Stefan Behnel wrote:
> Well - what do you need it for?

I need to manually create such objects using _PyLong_New() and then
manually fill in the ob_digit field. Currently, I use
(<PyLongObject*>L).ob_digit which is not a disaster, but it's not as
elegant as it could be.

> Cython is actually pretty good at optimising PyLong values, and fiddling
> with them manually is quite risky as their memory layout depends on a
> CPython compile time setting (digit size).

I'm using the constants defined in the Python headers (like
sizeof(digit) and PyLong_SHIFT). The Cython-generated code is compiled
using the Python header, so I don't see the problem here.

[Stefan Behnel]

Jeroen Demeyer schrieb am 26.02.2015 um 08:24:
> On 2015-02-25 15:15, Stefan Behnel wrote:
>> Well - what do you need it for?
> I need to manually create such objects using _PyLong_New() and then
> manually fill in the ob_digit field. Currently, I use
> (<PyLongObject*>L).ob_digit which is not a disaster, but it's not as
> elegant as it could be.

And would you mind sharing with us why you need to create the objects in
this unusual way?

>> Cython is actually pretty good at optimising PyLong values, and fiddling
>> with them manually is quite risky as their memory layout depends on a
>> CPython compile time setting (digit size).
> I'm using the constants defined in the Python headers (like sizeof(digit)
> and PyLong_SHIFT). The Cython-generated code is compiled using the Python
> header, so I don't see the problem here.

The problem is that the size of "digit" can either be specified by
configure, or determined at C compile time by the header files. I've seen
cases where the latter yielded a different size for me than what CPython
was really using internally, thus leading to crashes and/or data corruption.

Basically, if PYLONG_BITS_IN_DIGIT is defined by pyconfig.h, then you're on
the safe side. If it's not (and that's the normal case), then there's a
relatively high chance of being lucky, and a certain risk of being unlucky.

Stefan

[Jeroen Demeyer]

On 2015-02-26 08:54, Stefan Behnel wrote:
> Jeroen Demeyer schrieb am 26.02.2015 um 08:24:
>> On 2015-02-25 15:15, Stefan Behnel wrote:
>>> Well - what do you need it for?
>> I need to manually create such objects using _PyLong_New() and then
>> manually fill in the ob_digit field. Currently, I use
>> (<PyLongObject*>L).ob_digit which is not a disaster, but it's not as
>> elegant as it could be.
>
> And would you mind sharing with us why you need to create the objects in
> this unusual way?

To implement a conversion between GMP mpz_t and Python longs:
http://trac.sagemath.org/ticket/17853

> The problem is that the size of "digit" can either be specified by
> configure, or determined at C compile time by the header files.

That shouldn't matter. What matters is that the size of "digit" is a
deterministic function of the Python header files.

> I've seen
> cases where the latter yielded a different size for me than what CPython
> was really using internally, thus leading to crashes and/or data corruption.

How can this happen? Doesn't CPython use its own header files??? Perhaps
this is a bug which got fixed?

[Stefan Behnel]

Jeroen Demeyer schrieb am 26.02.2015 um 10:39:
> On 2015-02-26 08:54, Stefan Behnel wrote:
>> Jeroen Demeyer schrieb am 26.02.2015 um 08:24:
>>> On 2015-02-25 15:15, Stefan Behnel wrote:
>>>> Well - what do you need it for?
>>> I need to manually create such objects using _PyLong_New() and then
>>> manually fill in the ob_digit field. Currently, I use
>>> (<PyLongObject*>L).ob_digit which is not a disaster, but it's not as
>>> elegant as it could be.
>>
>> And would you mind sharing with us why you need to create the objects in
>> this unusual way?
> To implement a conversion between GMP mpz_t and Python longs:
> http://trac.sagemath.org/ticket/17853

Ok, got it. That's a very special use case that might call for some very
special measures.

Would _PyLong_FromByteArray() be an option?

>> The problem is that the size of "digit" can either be specified by
>> configure, or determined at C compile time by the header files.
> That shouldn't matter. What matters is that the size of "digit" is a
> deterministic function of the Python header files.
>
>> I've seen
>> cases where the latter yielded a different size for me than what CPython
>> was really using internally, thus leading to crashes and/or data corruption.
> How can this happen? Doesn't CPython use its own header files??? Perhaps
> this is a bug which got fixed?

https://bugs.python.org/issue22313

As to why this happens, I can't say. Would need some deeper digging than I
wanted to invest at the time.

Stefan

[Jeroen Demeyer]

On 2015-02-26 11:40, Stefan Behnel wrote:
> Would _PyLong_FromByteArray() be an option?

I think so, but that would be less efficient since there would be a
double conversion
PyLong <-> unsigned char* <-> mpz_t
instead of a direct conversion
PyLong <-> mpz_t

[Stefan Behnel]

Stefan Behnel schrieb am 26.02.2015 um 11:40:
> Jeroen Demeyer schrieb am 26.02.2015 um 10:39:
>> On 2015-02-26 08:54, Stefan Behnel wrote:
>>> Jeroen Demeyer schrieb am 26.02.2015 um 08:24:
>>>> I need to manually create such objects using _PyLong_New() and then
>>>> manually fill in the ob_digit field. Currently, I use
>>>> (<PyLongObject*>L).ob_digit which is not a disaster, but it's not as
>>>> elegant as it could be.
>>>

>>> The problem is that the size of "digit" can either be specified by
>>> configure, or determined at C compile time by the header files.
>> That shouldn't matter. What matters is that the size of "digit" is a
>> deterministic function of the Python header files.
>>
>>> I've seen
>>> cases where the latter yielded a different size for me than what CPython
>>> was really using internally, thus leading to crashes and/or data corruption.
>> How can this happen? Doesn't CPython use its own header files??? Perhaps
>> this is a bug which got fixed?
>
> https://bugs.python.org/issue22313
>
> As to why this happens, I can't say. Would need some deeper digging than I
> wanted to invest at the time.

I tried to reproduce this problem now and failed. So I'll just assume it
generally works and enabled the PyLong optimisations for all installations,
also for Py2.7 (it was previously limited to Py3.x). While not helping in
your use case, it should speed up arithmetic/binary operations with C long
sized Python integers in Cython, as long as one of them is a constant (so
that Cython can guess what the other one will also be a number).

Stefan