Re: Support for numpy.int64 arrays

[Ralph Heinkel]

Honestly I haven't tried 64bit arrays yet. My development machine was 32bit, so I have yet to give this a push.

I'm currently in a refactoring process with pyRserve, and so I'll put 64bit support on my TODO list.

Thanks for reporting,

Ralph

Am Montag, 18. Juni 2012 21:49:00 UTC+2 schrieb Vebjorn Ljosa:

It appears that pyRserve 1.5.2 cannot handle 64-bit integer numpy arrays:

$ python

Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53) 

[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> import numpy as np

>>> import pyRserve

>>> pyRserve.__version__

'0.5.2'

>>> conn = pyRserve.connect()

>>> a = np.array([[1,2,3],[4,5,6]], dtype='i8')

>>> conn.r.a = a

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rconn.py", line 186, in __setattr__

    self._rconn.setRexp(name, o)

  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rconn.py", line 32, in decoCheckIfClosed

    return func(self, *args, **kw)

  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rconn.py", line 122, in setRexp

    rAssign(name, o, self.sock)

  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rserializer.py", line 282, in rAssign

    s.serialize(o, dtTypeCode=rtypes.DT_SEXP)

  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rserializer.py", line 108, in serialize

    length = self._serializeExpr(o)

  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rserializer.py", line 118, in _serializeExpr

    rTypeHint = rtypes.numpyMap[o.dtype.type]   #o.dtype.type

KeyError: <type 'numpy.int64'>

>>>

Thanks,

Vebjorn

[Aidan Skinner]

On Monday, June 18, 2012 8:49:00 PM UTC+1, Vebjorn Ljosa wrote:

It appears that pyRserve 1.5.2 cannot handle 64-bit integer numpy arrays:

I worked around this on OS X with the following horid hack to force them to 32bit:

--- rserializer.py.orig 2012-06-20 14:47:02.901445715 +0100

+++ rserializer.py      2012-06-20 14:47:29.413448105 +0100

@@ -115,6 +115,8 @@

     def _serializeExpr(self, o, rTypeHint=None):

         if not rTypeHint:

             if isinstance(o, numpy.ndarray):

+                if o.dtype.type == numpy.int64:

+                    o = o.astype(numpy.int32)

                 rTypeHint = rtypes.numpyMap[o.dtype.type]   #o.dtype.type

             else:

                 rTypeHint = type(o)

[Ralph Heinkel]

Hi everybody,

I just had contact with Simon Urban, the author of Rserve (the server part that pyRserve connects to).
He states that R does not support 64 bit integers at all, which is why Rserve and hence pyRserve have no 64bit integer support. His suggestion was to convert 64bit integer arrays into double-float arrays before sending them to R(serve).

For that reason there will not be any direct 64bit integer support in pyRserve.

The solution Aidan proposed of automatically converting 64bit arrays to 32bit in pyRserve is pretty dangerous for the innocent user, so for the moment I won't implement this.
What I will do however is to provide a proper exception if a 64bit array is pushed into pyRserve.

Ciao ciao

Ralph

Am Montag, 18. Juni 2012 21:49:00 UTC+2 schrieb Vebjorn Ljosa:

It appears that pyRserve 1.5.2 cannot handle 64-bit integer numpy arrays:

$ python

Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53) 

[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> import numpy as np

>>> import pyRserve

>>> pyRserve.__version__

'0.5.2'

>>> conn = pyRserve.connect()

>>> a = np.array([[1,2,3],[4,5,6]], dtype='i8')

>>> conn.r.a = a

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rconn.py", line 186, in __setattr__

    self._rconn.setRexp(name, o)

  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rconn.py", line 32, in decoCheckIfClosed

    return func(self, *args, **kw)

  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rconn.py", line 122, in setRexp

    rAssign(name, o, self.sock)

  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rserializer.py", line 282, in rAssign

    s.serialize(o, dtTypeCode=rtypes.DT_SEXP)

  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rserializer.py", line 108, in serialize

    length = self._serializeExpr(o)

  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rserializer.py", line 118, in _serializeExpr

    rTypeHint = rtypes.numpyMap[o.dtype.type]   #o.dtype.type

KeyError: <type 'numpy.int64'>

>>>

Thanks,

Vebjorn

[Aidan Skinner]

Yeah, that patch is clearly suboptimal in a number of ways. 

It'd be good if we could figure out a way to do this nicely - maybe coercing int if they're within 32 bit range and throwing if otherwise? 

- Aidan