compiler warning - pointer difference assumed to be int

[Josh Ayers]

When compiling under Windows using 64-bit Visual Studio 2008, I get a C compiler warning associated with line 798 of numpy.pxd.  It complains that an __int64 is being converted to a long.  Here's the relevant section of numpy.pxd (with a few unimportant lines removed for clarity).

cdef inline char* _util_dtypestring(dtype descr, char* f, char* end, int* offset) except NULL:

    cdef dtype child
    cdef tuple i
    cdef tuple fields

    for childname in descr.names:
        fields = descr.fields[childname]
        child, new_offset = fields

        if (end - f) - (new_offset - offset[0]) < 15: # compiler warning on this line

Looking at the generated C code, PyInt_FromLong is called on (end - f), which is the difference of two pointers.  If I manually change the conversion function to PyInt_FromSsize_t, the warning goes away.

I believe the root problem is that Cython assumes a difference of two pointers can be stored in an int.  Here's another short example that gives the same warning. 

def ptr_diff():
    cdef int *a, *b
    diff = a - b # compiler warning on this line

    return diff

The diff variable is automatically typed by Cython as an int, while as far as I can tell, the compiler treats (a - b) as an __int64.

For reference, I'm using Python 2.7 and Cython 0.19.1.

Thanks,
Josh Ayers

[Josh Ayers]

Tangentially related, I just submitted a pull request that fixes three other MS compiler warnings, all associated with memoryviews.

https://github.com/cython/cython/pull/231

[Lawrence Mitchell]

Josh Ayers wrote:

[...]

> Looking at the generated C code, PyInt_FromLong is called on (end - f),
> which is the difference of two pointers. If I manually change the
> conversion function to PyInt_FromSsize_t, the warning goes away.

Ssize_t isn't necessarily large enough to hold the difference of
two pointers.

> I believe the root problem is that Cython assumes a difference of two
> pointers can be stored in an int. Here's another short example that gives
> the same warning.

> def ptr_diff():
> cdef int *a, *b
> diff = a - b # compiler warning on this line
> return diff

FWIW, the correct type for storing a pointer difference is
ptrdiff_t.


--
Lawrence Mitchell <we...@gmx.li>

[Josh Ayers]

> Looking at the generated C code, PyInt_FromLong is called on (end - f),
> which is the difference of two pointers.  If I manually change the
> conversion function to PyInt_FromSsize_t, the warning goes away.

Ssize_t isn't necessarily large enough to hold the difference of
two pointers.

Right, but to convert it to a Python integer, that function is the only reasonable option that I can find.
 

> I believe the root problem is that Cython assumes a difference of two
> pointers can be stored in an int.  Here's another short example that gives
> the same warning.

> def ptr_diff():
>     cdef int *a, *b
>     diff = a - b # compiler warning on this line
>     return diff

FWIW, the correct type for storing a pointer difference is
ptrdiff_t.

 

Is ptrdiff_t always available?  If so, then that seems like the right choice.  Either way, my main point was that Cython should infer something other than an int, which is incorrect on any 64 bit system.

[Robert Bradshaw]

On Tue, May 21, 2013 at 9:32 AM, Josh Ayers <josh....@gmail.com> wrote:
>
>>
>> > Looking at the generated C code, PyInt_FromLong is called on (end - f),
>> > which is the difference of two pointers. If I manually change the
>> > conversion function to PyInt_FromSsize_t, the warning goes away.
>>
>> Ssize_t isn't necessarily large enough to hold the difference of
>> two pointers.
>
>
> Right, but to convert it to a Python integer, that function is the only
> reasonable option that I can find.

I don't know of any modern systems where ptrdiff_t != ssize_t, though
it is allowed by the C standard.

>> > I believe the root problem is that Cython assumes a difference of two
>> > pointers can be stored in an int. Here's another short example that
>> > gives
>> > the same warning.
>>
>> > def ptr_diff():
>> > cdef int *a, *b
>> > diff = a - b # compiler warning on this line
>> > return diff
>>
>> FWIW, the correct type for storing a pointer difference is
>> ptrdiff_t.
>
>
> Is ptrdiff_t always available? If so, then that seems like the right
> choice. Either way, my main point was that Cython should infer something
> other than an int, which is incorrect on any 64 bit system.

It's part of ANSI C (89), so it should be.

https://github.com/cython/cython/commit/3087d7fd4f78514f3e962a321b5c865dafe859d9
https://github.com/cython/cython/commit/1fcaf85398e4a3ee9d4d77fc0647fdf9984b1d5b

[Josh Ayers]

> Is ptrdiff_t always available?  If so, then that seems like the right
> choice.  Either way, my main point was that Cython should infer something
> other than an int, which is incorrect on any 64 bit system.

It's part of ANSI C (89), so it should be.

https://github.com/cython/cython/commit/3087d7fd4f78514f3e962a321b5c865dafe859d9
https://github.com/cython/cython/commit/1fcaf85398e4a3ee9d4d77fc0647fdf9984b1d5b

Awesome, thanks for the fix.  As I suspected, this one was over my head.

If you don't mind the noob question, can I ask why you used PyInt_FromLong instead of PyInt_FromSsize in __Pyx_PyInt_FromPtrdiff_t?  Is it because there is no SSIZE_MIN and _MAX that would let you check when it's within range for the latter function?

Thanks again,
Josh

[Robert Bradshaw]

In Py2, there are two integer types, one for values that fit in a long
(called int) and the other for arbitrary precision values (long). The
former are faster to construct and work with.

- Robert