recv_into(bytearray) complains about a "pinned buffer"
Andrew Dalke
TypeError which complains about a "pinned buffer". I have only an
inkling of what that means. Since an array.array("b") works there, and
since it works in Python 3.1.1, and since I thought the point of a
bytearray was to make things like recv_into easier, I think this
exception is a bug in Python 2.6.
I want to double check before posting it to the tracker.
Here's my reproducibles:
Python 2.6.1 (r261:67515, Jul 7 2009, 23:51:51)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> sock = socket.socket()
>>> sock.connect( ("python.org", 80) )
>>> sock.send(b"GET / HTTP/1.0\r\n\r\n")
18
>>> buf = bytearray(b" " * 10)
>>> sock.recv_into(buf)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: recv_into() argument 1 must be pinned buffer, not bytearray
>>>
I expected a bytearray to work there. In fact, I thought the point of
bytearray was to allow this to work.
By comparison, an array of bytes does work:
>>> import array
>>> arr = array.array("b")
>>> arr.extend(map(ord, "This is a test"))
>>> len(arr)
14
>>> sock.recv_into(arr)
14
>>> arr
array('b', [72, 84, 84, 80, 47, 49, 46, 49, 32, 51, 48, 50, 32, 70])
>>> "".join(map(chr, arr))
'HTTP/1.1 302 F'
I don't even know what a "pinned buffer" means, and searching
python.org isn't helpful.
Using a bytearray in Python 3.1.1 *does* work:
Python 3.1.1 (r311:74480, Jan 31 2010, 23:07:16)
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> sock = socket.socket()
>>> sock.connect( ("python.org", 80) )
>>> sock.send(b"GET / HTTP/1.0\r\n\r\n")
18
>>> buf = bytearray(b" " * 10)
>>> sock.recv_into(buf)
10
>>> buf
bytearray(b'HTTP/1.1 3')
>>>
Is this a bug in Python 2.6 or a deliberate choice regarding
implementation concerns I don't know about?
If it's a bug, I'll add it to the tracker.
Andrew Dalke
da...@dalkescientific.com
Antoine Pitrou
Hello Andrew,
> I don't even know what a "pinned buffer" means, and searching python.org
> isn't helpful.
>
> Using a bytearray in Python 3.1.1 *does* work:
> [...]
Agreed, the error message is cryptic.
The problem is that socket.recv_into() in 2.6 doesn't recognize the new
buffer API which is needed to accept bytearray objects.
(it does in 3.1, because the old buffer API doesn't exist anymore there)
You could open an issue on the bug tracker for this.
Thank you
Antoine.
Andrew Dalke
> The problem is that socket.recv_into() in 2.6 doesn't recognize the new
> buffer API which is needed to accept bytearray objects.
> (it does in 3.1, because the old buffer API doesn't exist anymore there)
That's about what I thought it was, but I don't know if this was a
deliberate choice or accidental.
BTW, 2.7 (freshly built from version control) also has the same
exception.
> You could open an issue on the bug tracker for this.
I've done that. It's http://bugs.python.org/issue7827 .
Cheers!
Andrew
da...@dalkescientific.com
Martin v. Loewis
> TypeError which complains about a "pinned buffer". I have only an
> inkling of what that means.
A pinned buffer is one that cannot move in memory, even if another
thread tries to behind your back. Typically, resizable containers
are not inherently pinned, and "a user" (i.e. the API function) must
explicitly pin it, which recv_into fails to do.
> Is this a bug in Python 2.6 or a deliberate choice regarding
> implementation concerns I don't know about?
It's actually a bug also that you pass an array; doing so *should*
give the very same error.
It may be that the bugs surrounding buffers will never get fully
resolved in the lifetime of Python 2.x, so I would probably just ignore
recv_into.
Regards,
Martin
Antoine Pitrou
>
>> Is this a bug in Python 2.6 or a deliberate choice regarding
>> implementation concerns I don't know about?
>
> It's actually a bug also that you pass an array; doing so *should* give
> the very same error.
Well, if you can give neither an array nor a bytearray to recv_into(),
what *could* you give it?
recv_into() should simply be fixed to use the new buffer API, as it does
in 3.x.
Martin v. Loewis
> Le Mon, 01 Feb 2010 03:30:56 +0100, Martin v. Loewis a écrit :
>>> Is this a bug in Python 2.6 or a deliberate choice regarding
>>> implementation concerns I don't know about?
>> It's actually a bug also that you pass an array; doing so *should* give
>> the very same error.
>
> Well, if you can give neither an array nor a bytearray to recv_into(),
> what *could* you give it?
My recommendation would be to not use recv_into in 2.x, but only in 3.x.
> recv_into() should simply be fixed to use the new buffer API, as it does
> in 3.x.
I don't think that's the full solution. The array module should also
implement the new buffer API, so that it would also fail with the old
recv_into.
Regards,
Martin
Andrew Dalke
> My recommendation would be to not use recv_into in 2.x, but only in 3.x.
> I don't think that's the full solution. The array module should also
> implement the new buffer API, so that it would also fail with the old
> recv_into.
Okay. But recv_into was added in 2.5 and the test case in
2.6's test_socket.py clearly allows an array there:
def testRecvInto(self):
buf = array.array('c', ' '*1024)
nbytes = self.cli_conn.recv_into(buf)
self.assertEqual(nbytes, len(MSG))
msg = buf.tostring()[:len(MSG)]
self.assertEqual(msg, MSG)
Checking koders and Google Code search engines, I found one project
which used recv_into, with the filename bmpreceiver.py . It
uses a array.array("B", [0] * length) .
Clearly it was added to work with an array, and it's
being used with an array. Why shouldn't people use it
with Python 2.x?
Andrew
da...@dalkescientific.com
Antoine Pitrou
>> recv_into() should simply be fixed to use the new buffer API, as it
>> does in 3.x.
>
> I don't think that's the full solution. The array module should also
> implement the new buffer API, so that it would also fail with the old
> recv_into.
There was a patch for this in http://bugs.python.org/issue6071 , but the
bug was closed when hashlib was "fixed".
Martin v. Loewis
> being used with an array. Why shouldn't people use it
> with Python 2.x?
Because it's not thread-safe; it may crash the interpreter if used
incorrectly.
Of course, if you don't share the array across threads, it can be safe
to use.
Regards,
Martin