faster gmpy2

[20100.d...@gmail.com]

Dear all,

We should soon see a release of gmpy with a C-API and cython header (gmpy is a Python wrapper for GMP, MPFR and MPC [1]). All the Cython interface is contained in the header gmpy2.pxd [2]. I would like tighter Cython integration and am asking for comments about the feasability of

A) to be able to use `cdef mpz a` (right now I can only cast `object` to `MPZ_Object *` when I want to use the gmpy2 C-API)

B) if `a`, `b`, `c` are mpz objects, I would like `c = a + b` to be translated directly into `mpz_add(MPZ(<MPZ_Object *> c), MPZ(<MPZ_Object *> a), MPZ(<MPZ_Object *> b))` in order to avoid the lengthy python calls.

Thanks
Vincent

[1] https://github.com/aleaxit/gmpy
[2] https://github.com/aleaxit/gmpy/blob/master/src/gmpy2.pxd

[casevh]

On Friday, July 14, 2017 at 3:53:02 PM UTC-7, 20100.d...@gmail.com wrote:

Dear all,

We should soon see a release of gmpy with a C-API and cython header (gmpy is a Python wrapper for GMP, MPFR and MPC [1]). All the Cython interface is contained in the header gmpy2.pxd [2]. I would like tighter Cython integration and am asking for comments about the feasability of

A) to be able to use `cdef mpz a` (right now I can only cast `object` to `MPZ_Object *` when I want to use the gmpy2 C-API)

B) if `a`, `b`, `c` are mpz objects, I would like `c = a + b` to be translated directly into `mpz_add(MPZ(<MPZ_Object *> c), MPZ(<MPZ_Object *> a), MPZ(<MPZ_Object *> b))` in order to avoid the lengthy python calls.

Hi Vincent,

I'm the gmpy2 maintainer and I really should be working on getting the release finished....

I don't know if this will help, but I could expose additional gmpy2 functions via the C-API. For example, GMPy_Number_Add() will add any two native Python or gmpy2 types. It also accepts an optional context argument to control rounding, precision, manage exceptions, etc. Your example (ignoring Cython syntax) would translate into `c=GMPy_Number_Add(a,b,NULL)`. As currently written, it will only recognize native Python or gmpy2 types but the logic could be expanded to include checks for __mpz_, __mpq__, __mpfr__, and __mpc__ methods and automatically convert the objects.

`gmpy2` maintains a free list of objects so there is minimal overhead for create the returned object. This approach may not be as fast as directly calling GMP/MPFR/MPC but it might be a simpler translation of Python/Cython code. 

If `gmpy2` is statically linked, it will eliminate the need for local copies of GMP/MPFR/MPC. This would be very helpful for Windows users.

Thanks, Vincent, for helping add the C-API!

Regards,

Case

[Stefan Behnel]

20100.d...@gmail.com schrieb am 15.07.2017 um 00:53:
> We should soon see a release of gmpy with a C-API and cython header (gmpy
> is a Python wrapper for GMP, MPFR and MPC [1]). All the Cython interface is
> contained in the header gmpy2.pxd [2].
>

> [1] https://github.com/aleaxit/gmpy
> [2] https://github.com/aleaxit/gmpy/blob/master/src/gmpy2.pxd

Nice.

> I would like tighter Cython
> integration and am asking for comments about the feasability of
>
> A) to be able to use `cdef mpz a` (right now I can only cast `object` to
> `MPZ_Object *` when I want to use the gmpy2 C-API)

Should be possible. See

http://docs.cython.org/en/latest/src/userguide/extension_types.html#public-and-external-extension-types

> B) if `a`, `b`, `c` are mpz objects, I would like `c = a + b` to be
> translated directly into `mpz_add(MPZ(<MPZ_Object *> c), MPZ(<MPZ_Object *>
> a), MPZ(<MPZ_Object *> b))` in order to avoid the lengthy python calls.

Not easily. Something similar was just implemented for NumPy array
expressions, using Pythran as a backend, but that's fairly specialised.
There should probably be some C-level protocol for expressions at some
point (something like "cdef inline __add__(self, other)"), but there
currently isn't.

Stefan

[20100.d...@gmail.com]

Le dimanche 16 juillet 2017 17:29:43 UTC+2, Stefan Behnel a écrit :

20100.d...@gmail.com schrieb am 15.07.2017 um 00:53:

> I would like tighter Cython
> integration and am asking for comments about the feasability of
>
> A) to be able to use `cdef mpz a` (right now I can only cast `object` to
> `MPZ_Object *` when I want to use the gmpy2 C-API)

Should be possible. See

http://docs.cython.org/en/latest/src/userguide/extension_types.html#public-and-external-extension-types

Wonderful! It is now way much cleaner

    https://github.com/videlec/gmpy/blob/cleaner_cython/src/gmpy2.pxd

However I am facing with a test failure now. The innocent "assert x == 84 + 60j" at line 201 of

    https://github.com/videlec/gmpy/blob/cleaner_cython/test_cython/test_cython.pyx

gives me

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "test_cython.pyx", line 355, in init test_cython (test_cython.c:6681)
    test_mpc()
  File "test_cython.pyx", line 201, in test_cython.test_mpc (test_cython.c:3344)
    assert x == 84 + 60j
TypeError: Cannot convert complex to mpc

The same kind of comparisons appear in other places in the file test_cython.pyx. Any idea about what can go wrong? I am using Python 3.6.1 and Cython 0.25.2.
 

> B) if `a`, `b`, `c` are mpz objects, I would like `c = a + b` to be
> translated directly into `mpz_add(MPZ(<MPZ_Object *> c), MPZ(<MPZ_Object *>
> a), MPZ(<MPZ_Object *> b))` in order to avoid the lengthy python calls.

Not easily. Something similar was just implemented for NumPy array
expressions, using Pythran as a backend, but that's fairly specialised.
There should probably be some C-level protocol for expressions at some
point (something like "cdef inline __add__(self, other)"), but there
currently isn't.

Would be really cool!

[20100.d...@gmail.com]

Le dimanche 16 juillet 2017 19:23:43 UTC+2, 20100.d...@gmail.com a écrit :

Le dimanche 16 juillet 2017 17:29:43 UTC+2, Stefan Behnel a écrit :

20100.d...@gmail.com schrieb am 15.07.2017 um 00:53:

The same kind of comparisons appear in other places in the file test_cython.pyx.

Let me be more precise: such comparisons appear in other places and run without any problem!

[Stefan Behnel]

20100.d...@gmail.com schrieb am 16.07.2017 um 19:23:
> I am facing with a test failure now. The innocent "assert x == 84 + 60j"
> at line 201 of
>
> https://github.com/videlec/gmpy/blob/cleaner_cython/test_cython/test_cython.pyx
>
> gives me
>
> Traceback (most recent call last):
> File "<string>", line 1, in <module>
> File "test_cython.pyx", line 355, in init test_cython (test_cython.c:6681)
> test_mpc()
> File "test_cython.pyx", line 201, in test_cython.test_mpc
> (test_cython.c:3344)
> assert x == 84 + 60j
> TypeError: Cannot convert complex to mpc
>
> The same kind of comparisons appear in other places in the file
> test_cython.pyx. Any idea about what can go wrong? I am using Python 3.6.1
> and Cython 0.25.2.

The other places all use a "del x" for that variable. Does it work if you
add it to the failing test, too? And do the others fail as well when you
remove it there?

Stefan

[20100.d...@gmail.com]

I had a mistake in the test file so that some tests were not run.

- All the tests relative to comparison between mpc and complex do fail (with or without del statements)
- All the others involving mpz, mpq, mpfr are fine

[20100.d...@gmail.com]

I believe it is a problem on Cython side. Any comparison involving gmpy2 types should go through the function GMPy_RichCompare_Slot in the file

    https://github.com/videlec/gmpy/blob/cleaner_cython/src/gmpy2_richcompare.c

However, when comparing mpc with complex (in Cython only) the function GMPy_RichCompare_Slot is not called.

[20100.d...@gmail.com]

The following snippet
{{{
cdef mpc x = GMPy_MPC_New(56, 56, NULL)
x == 0
x == 0.0
x == 0.0j
}}}
produces the following error on the last line
{{{

Traceback (most recent call last):
  File "<string>", line 1, in <module>

  File "test_mpc.pyx", line 13, in init test_mpc (test_mpc.c:1536)
    x == 0.0j

TypeError: Cannot convert complex to mpc
}}}

The type check before the comparison produced by Cython for "x == 0.0j"  looks weird to me
{{{
  /* "test_mpc.pyx":8
 * cdef mpc x = GMPy_MPC_New(56, 56, NULL)
 * 
 * x == 0             # <<<<<<<<<<<<<<
 * x == 0.0
 * x == 0.0j
 */
  __pyx_t_1 = PyObject_RichCompare(((PyObject *)__pyx_v_8test_mpc_x), __pyx_int_0, Py_EQ); __Pyx_XGOTREF(__pyx_t_1); if (unlikely(!__pyx_t_1)) __PYX_ERR(1, 8, __pyx_L1_error)
  __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
 
  /* "test_mpc.pyx":9
 * 
 * x == 0
 * x == 0.0             # <<<<<<<<<<<<<<
 * x == 0.0j
 */
  __pyx_t_1 = PyObject_RichCompare(((PyObject *)__pyx_v_8test_mpc_x), __pyx_float_0_0, Py_EQ); __Pyx_XGOTREF(__pyx_t_1); if (unlikely(!__pyx_t_1)) __PYX_ERR(1, 9, __pyx_L1_error)
  __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
 
  /* "test_mpc.pyx":10
 * x == 0
 * x == 0.0
 * x == 0.0j             # <<<<<<<<<<<<<<
 */
  __pyx_t_1 = PyComplex_FromDoubles(0.0, 0.0); if (unlikely(!__pyx_t_1)) __PYX_ERR(1, 10, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_1);
  if (!(likely(__Pyx_TypeTest(__pyx_t_1, __pyx_ptype_5gmpy2_mpc)))) __PYX_ERR(1, 10, __pyx_L1_error)
  __pyx_t_2 = PyObject_RichCompare(((PyObject *)__pyx_v_8test_mpc_x), __pyx_t_1, Py_EQ); __Pyx_XGOTREF(__pyx_t_2); if (unlikely(!__pyx_t_2)) __PYX_ERR(1, 10, __pyx_L1_error)
  __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
}}}

[Stefan Behnel]

Thanks for investigating. It's most likely this place that generates that
type check:

https://github.com/cython/cython/blob/6030d91cc73f0f09020f18ffa05181b28020335f/Cython/Compiler/ExprNodes.py#L888

This is correct in the case of an assignment but not for a comparison, so
my guess is that this is probably not the right place to fix this problem.
Needs some more looking into.

Stefan

[20100.d...@gmail.com]

Issue opened on github

    https://github.com/cython/cython/issues/1776

[Stefan Behnel]

20100.d...@gmail.com schrieb am 17.07.2017 um 16:39:
> Issue opened on github
>
> https://github.com/cython/cython/issues/1776

Fix is here:

https://github.com/cython/cython/pull/1777

Stefan

[20100.d...@gmail.com]

Thanks!