Re: Support for numpy.int64 arrays

141 views
Skip to first unread message

Ralph Heinkel

unread,
Jun 19, 2012, 4:51:35 AM6/19/12
to pyrs...@googlegroups.com
Honestly I haven't tried 64bit arrays yet. My development machine was 32bit, so I have yet to give this a push.

I'm currently in a refactoring process with pyRserve, and so I'll put 64bit support on my TODO list.

Thanks for reporting,

Ralph

Am Montag, 18. Juni 2012 21:49:00 UTC+2 schrieb Vebjorn Ljosa:
It appears that pyRserve 1.5.2 cannot handle 64-bit integer numpy arrays:

$ python
Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53) 
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> import pyRserve
>>> pyRserve.__version__
'0.5.2'
>>> conn = pyRserve.connect()
>>> a = np.array([[1,2,3],[4,5,6]], dtype='i8')
>>> conn.r.a = a
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rconn.py", line 186, in __setattr__
    self._rconn.setRexp(name, o)
  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rconn.py", line 32, in decoCheckIfClosed
    return func(self, *args, **kw)
  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rconn.py", line 122, in setRexp
    rAssign(name, o, self.sock)
  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rserializer.py", line 282, in rAssign
    s.serialize(o, dtTypeCode=rtypes.DT_SEXP)
  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rserializer.py", line 108, in serialize
    length = self._serializeExpr(o)
  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rserializer.py", line 118, in _serializeExpr
    rTypeHint = rtypes.numpyMap[o.dtype.type]   #o.dtype.type
KeyError: <type 'numpy.int64'>
>>>

Thanks,
Vebjorn

Aidan Skinner

unread,
Jun 20, 2012, 9:54:32 AM6/20/12
to pyrs...@googlegroups.com
On Monday, June 18, 2012 8:49:00 PM UTC+1, Vebjorn Ljosa wrote:
It appears that pyRserve 1.5.2 cannot handle 64-bit integer numpy arrays:

I worked around this on OS X with the following horid hack to force them to 32bit:

--- rserializer.py.orig 2012-06-20 14:47:02.901445715 +0100
+++ rserializer.py      2012-06-20 14:47:29.413448105 +0100
@@ -115,6 +115,8 @@
     def _serializeExpr(self, o, rTypeHint=None):
         if not rTypeHint:
             if isinstance(o, numpy.ndarray):
+                if o.dtype.type == numpy.int64:
+                    o = o.astype(numpy.int32)
                 rTypeHint = rtypes.numpyMap[o.dtype.type]   #o.dtype.type
             else:
                 rTypeHint = type(o)

Ralph Heinkel

unread,
Jun 20, 2012, 11:04:58 AM6/20/12
to pyrs...@googlegroups.com
Hi everybody,

I just had contact with Simon Urban, the author of Rserve (the server part that pyRserve connects to).
He states that R does not support 64 bit integers at all, which is why Rserve and hence pyRserve have no 64bit integer support. His suggestion was to convert 64bit integer arrays into double-float arrays before sending them to R(serve).

For that reason there will not be any direct 64bit integer support in pyRserve.

The solution Aidan proposed of automatically converting 64bit arrays to 32bit in pyRserve is pretty dangerous for the innocent user, so for the moment I won't implement this.
What I will do however is to provide a proper exception if a 64bit array is pushed into pyRserve.

Ciao ciao


Ralph

Am Montag, 18. Juni 2012 21:49:00 UTC+2 schrieb Vebjorn Ljosa:
It appears that pyRserve 1.5.2 cannot handle 64-bit integer numpy arrays:

$ python
Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53) 
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> import pyRserve
>>> pyRserve.__version__
'0.5.2'
>>> conn = pyRserve.connect()
>>> a = np.array([[1,2,3],[4,5,6]], dtype='i8')
>>> conn.r.a = a
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rconn.py", line 186, in __setattr__
    self._rconn.setRexp(name, o)
  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rconn.py", line 32, in decoCheckIfClosed
    return func(self, *args, **kw)
  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rconn.py", line 122, in setRexp
    rAssign(name, o, self.sock)
  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rserializer.py", line 282, in rAssign
    s.serialize(o, dtTypeCode=rtypes.DT_SEXP)
  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rserializer.py", line 108, in serialize
    length = self._serializeExpr(o)
  File "/Users/ljosa/local/lib/python2.7/site-packages/pyRserve/rserializer.py", line 118, in _serializeExpr
    rTypeHint = rtypes.numpyMap[o.dtype.type]   #o.dtype.type
KeyError: <type 'numpy.int64'>
>>>

Thanks,
Vebjorn

Aidan Skinner

unread,
Jun 20, 2012, 12:51:57 PM6/20/12
to pyrs...@googlegroups.com
Yeah, that patch is clearly suboptimal in a number of ways. 

It'd be good if we could figure out a way to do this nicely - maybe coercing int if they're within 32 bit range and throwing if otherwise? 

- Aidan 
Reply all
Reply to author
Forward
0 new messages