How to create a socket.socket() object from a socket fd?

[Grant Edwards]

Given a Unix file discriptor for an open TCP socket, I can't figure
out how to create a python 2.7 socket object like those returned by
socket.socket()

Based on the docs, one might think that socket.fromfd() would do that
(since the docs say that's what it does):

Quoting https://docs.python.org/2/library/socket.html

the socket() function returns a socket object

[...]

socket.socket([family[, type[, proto]]])
Create a new socket using the given address family[...]

[...]

socket.fromfd(fd, family, type[, proto])
Duplicate the file descriptor fd (an integer as returned by a file
object’s fileno() method) and build a socket object from the
result.

But it doesn't work as described -- the docs apprently use the term
"socket object" to refer to two different things.

IOW, a "socket object" is a different type than a "socket object".

Here's what a "socket object" returned by socket.socket() looks like:

repr(osock):
<socket._socketobject object at 0x7f701948f520>

dir(osock):
['__class__', '__delattr__', '__doc__', '__format__',
'__getattribute__', '__hash__', '__init__', '__module__', '__new__',
'__reduce__', '__reduce_ex__', '__repr__', '__setattr__',
'__sizeof__', '__slots__', '__str__', '__subclasshook__',
'__weakref__', '_sock', 'accept', 'bind', 'close', 'connect',
'connect_ex', 'dup', 'family', 'fileno', 'getpeername', 'getsockname',
'getsockopt', 'gettimeout', 'listen', 'makefile', 'proto', 'recv',
'recv_into', 'recvfrom', 'recvfrom_into', 'send', 'sendall', 'sendto',
'setblocking', 'setsockopt', 'settimeout', 'shutdown', 'type']

[If asked, I would call that a "socket._socketobject object".]

Here's what a "socket object" returned from socket.fromfd() object looks like:

repr(sock):
<socket object, fd=6, family=2, type=1, protocol=0>

dir(sock):
['__class__', '__delattr__', '__doc__', '__format__',
'__getattribute__', '__hash__', '__init__', '__new__', '__reduce__',
'__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__',
'__subclasshook__', 'accept', 'bind', 'close', 'connect',
'connect_ex', 'dup', 'family', 'fileno', 'getpeername', 'getsockname',
'getsockopt', 'gettimeout', 'listen', 'makefile', 'proto', 'recv',
'recv_into', 'recvfrom', 'recvfrom_into', 'send', 'sendall', 'sendto',
'setblocking', 'setsockopt', 'settimeout', 'shutdown', 'timeout',
'type']

They're different types.

[That one I sould all a "socket object".]

Note that the socket.socket() object has a '_sock' attribute, which
appears to contain an object like that returned by socket.fromfd():

repr(osock._sock):
<socket object, fd=4, family=2, type=1, protocol=0>

That prompts a question: given a "socket object" as returned by
socket.fromfd(), how does one create a "socket._socketobject object"
as returned by socket.socket()?

It also makes one wonder why doesn't socket.fromfd() return the same
type of object as socke.socket()? In what context would you want the
type of object returned from socket.fromrd() instead of that returned
by socket.socket()?

[An ssl context will wrap the latter, but not the former, in case
you're wondering why I care about the difference.]

Also, I passed socket.fromfd fd==5, and the resulting "socket object"
is using fd==6. Why does socket.fromfd() duplicate the fd? If I
wanted the file descriptor duplicated, I would have do it myself!
[Yes, I know the documentation _says_ it will duplicate the fd, I just
don't understand why, and I think it a bad idea.]

--
Grant

[Chris Angelico]

On Sun, Jan 22, 2017 at 9:28 AM, Grant Edwards
<grant.b...@gmail.com> wrote:
> Given a Unix file discriptor for an open TCP socket, I can't figure
> out how to create a python 2.7 socket object like those returned by
> socket.socket()

I suspect you can't easily do it. In more recent Pythons, you can
socket.socket(fileno=N), but that doesn't exist in 2.7. But maybe.....

> Here's what a "socket object" returned by socket.socket() looks like:
>
> repr(osock):
> <socket._socketobject object at 0x7f701948f520>
>

> Here's what a "socket object" returned from socket.fromfd() object looks like:
>
> repr(sock):
> <socket object, fd=6, family=2, type=1, protocol=0>
>

> Note that the socket.socket() object has a '_sock' attribute, which
> appears to contain an object like that returned by socket.fromfd():
>
> repr(osock._sock):
> <socket object, fd=4, family=2, type=1, protocol=0>

... maybe you could construct a socket.socket(), then monkeypatch its _sock??

> [An ssl context will wrap the latter, but not the former, in case
> you're wondering why I care about the difference.]
>
> Also, I passed socket.fromfd fd==5, and the resulting "socket object"
> is using fd==6. Why does socket.fromfd() duplicate the fd? If I
> wanted the file descriptor duplicated, I would have do it myself!
> [Yes, I know the documentation _says_ it will duplicate the fd, I just
> don't understand why, and I think it a bad idea.]

Agreed. It's much cleaner in Py3, but I don't know of a backport.

ChrisA

[Grant Edwards]

On 2017-01-21, Grant Edwards <grant.b...@gmail.com> wrote:

> Given a Unix file discriptor for an open TCP socket, I can't figure
> out how to create a python 2.7 socket object like those returned by
> socket.socket()
>

> Based on the docs, one might think that socket.fromfd() would do that
> (since the docs say that's what it does):

[...]

> That prompts a question: given a "socket object" as returned by
> socket.fromfd(), how does one create a "socket._socketobject object"
> as returned by socket.socket()?

Of course I figured it out immediately after spending 15 minutes
whinging about it.

help(socket.socket) gives you a hint:

class _socketobject(__builtin__.object)
| socket([family[, type[, proto]]]) -> socket object
|
[...]
|
| Methods defined here:
|
| __init__(self, family=2, type=1, proto=0, _sock=None)
|

Ah! There's a keyword argument that doesn't appear in the docs, so
let's try that...

context = ssl.create_default_context()
[...]
sock = socket.fromfd(fd2,socket.AF_INET,socket.SOCK_STREAM)
nsock = socket.socket(socket.AF_INET,socket.SOCK_STREAM,_sock=sock)
conn = context.wrap_socket(nsock, server_hostname="whatever.invalid")

That works.

--
Grant

[Chris Angelico]

On Sun, Jan 22, 2017 at 9:41 AM, Grant Edwards
<grant.b...@gmail.com> wrote:
> | __init__(self, family=2, type=1, proto=0, _sock=None)
> |
>
> Ah! There's a keyword argument that doesn't appear in the docs, so
> let's try that...

That's marginally better than my monkeypatch-after-creation
suggestion, but still broadly the same. Your code may well break in
other Python implementations, but within CPython 2.7, you should be
pretty safe.

ChrisA

[Christian Heimes]

On 2017-01-21 23:41, Grant Edwards wrote:
> On 2017-01-21, Grant Edwards <grant.b...@gmail.com> wrote:
>
>> Given a Unix file discriptor for an open TCP socket, I can't figure
>> out how to create a python 2.7 socket object like those returned by
>> socket.socket()
>>
>> Based on the docs, one might think that socket.fromfd() would do that
>> (since the docs say that's what it does):
> [...]
>> That prompts a question: given a "socket object" as returned by
>> socket.fromfd(), how does one create a "socket._socketobject object"
>> as returned by socket.socket()?
>
> Of course I figured it out immediately after spending 15 minutes
> whinging about it.
>
> help(socket.socket) gives you a hint:
>
> class _socketobject(__builtin__.object)
> | socket([family[, type[, proto]]]) -> socket object
> |
> [...]
> |
> | Methods defined here:
> |

> | __init__(self, family=2, type=1, proto=0, _sock=None)
> |
>
> Ah! There's a keyword argument that doesn't appear in the docs, so
> let's try that...
>

> context = ssl.create_default_context()
> [...]
> sock = socket.fromfd(fd2,socket.AF_INET,socket.SOCK_STREAM)
> nsock = socket.socket(socket.AF_INET,socket.SOCK_STREAM,_sock=sock)
> conn = context.wrap_socket(nsock, server_hostname="whatever.invalid")
>
> That works.

You might be interested in my small module
https://pypi.python.org/pypi/socketfromfd/ . I just releases a new
version with a fix for Python 2. Thanks for the hint! :)

The module correctly detects address family, socket type and proto from
a fd. It works correctly with e.g. IPv6 or Unix sockets. Ticket
https://bugs.python.org/issue28134 has additional background information
on the matter.

Christian

[Peter Otten]

Grant Edwards wrote:

> Given a Unix file discriptor for an open TCP socket, I can't figure
> out how to create a python 2.7 socket object like those returned by
> socket.socket()
>
> Based on the docs, one might think that socket.fromfd() would do that
> (since the docs say that's what it does):

[...]

> Note that the socket.socket() object has a '_sock' attribute, which
> appears to contain an object like that returned by socket.fromfd():
>
> repr(osock._sock):
> <socket object, fd=4, family=2, type=1, protocol=0>
>

> That prompts a question: given a "socket object" as returned by
> socket.fromfd(), how does one create a "socket._socketobject object"
> as returned by socket.socket()?

The socket.py source reveals that you can do it like this:

mysocket = socket.socket(_sock=socket.fromfd(...))

The code has

from _socket import socket # actually a star import
...
_realsocket = socket
...
socket = _socketobject

which looks very much like an ad-hoc hack gone permanent.

[Grant Edwards]

On 2017-01-21, Chris Angelico <ros...@gmail.com> wrote:
> On Sun, Jan 22, 2017 at 9:41 AM, Grant Edwards
><grant.b...@gmail.com> wrote:

>> | __init__(self, family=2, type=1, proto=0, _sock=None)
>> |
>>
>> Ah! There's a keyword argument that doesn't appear in the docs, so
>> let's try that...
>

> That's marginally better than my monkeypatch-after-creation
> suggestion, but still broadly the same. Your code may well break in
> other Python implementations, but within CPython 2.7, you should be
> pretty safe.

For those of you still paying attention...

It looks like CPython 3.4 socket.fromfd() does return the same type
object as socket.socket(). And, using the fileno= argument to
socket.socket() avoids duplicating the descriptor.

Looks like somebody with a time machine read my original post...

--
Grant

[Grant Edwards]

On 2017-01-21, Christian Heimes <chri...@python.org> wrote:

> You might be interested in my small module
> https://pypi.python.org/pypi/socketfromfd/ . I just releases a new
> version with a fix for Python 2. Thanks for the hint! :)
>
> The module correctly detects address family, socket type and proto from
> a fd. It works correctly with e.g. IPv6 or Unix sockets. Ticket
> https://bugs.python.org/issue28134 has additional background information
> on the matter.

Yes, thanks!

Just a few minutes ago I stumbled across that issue. For Python3, I
was using:

sock = socket.socket(fileno=fd)

But as you point out in that issue, the Python3 docs are wrong: when
using socket.socket(fileno=fd) you _do_ have to specify the correct
family and type parameters that correspond to the socket file
descriptor. So, I starting looking for os.getsockopt(), which doesn't
exist.

I see you use ctypes to call gestsockopt (that was going to be my next
step).

I suspect the code I'm working will end up being re-written in C for
the real product (so that it can run in-process in a thread rather
than as an external helper process). If not, I'll have to use your
module (or something like it) so that the solution will work on both
IPv4 and IPv6 TCP sockets (I'd also like it to work with Unix domain
sockets, but the piece at the other end of the socket connection
currently only supports TCP).

--
Grant

[Grant Edwards]

Newsgroups: gmane.comp.python.general
From: Grant Edwards <grant.b...@gmail.com>
Subject: Re: How to create a socket.socket() object from a socket fd?
References: <o60naq$bbm$1...@blaine.gmane.org> <o60o2r$6cd$1...@blaine.gmane.org> <c7e3116f-6e9c-5343...@python.org>
Followup-To:

<rant>

I'm still baffled why the standard library fromfd() code dup()s the
descriptor.

According to the comment in the CPython sources, the author of
fromfd() is guessing that the user wants to be able to close the
descriptor separately from the socket.

If the user wanted the socket object to use a duplicate descriptor for
some reason, the caller should call os.dup() -- it's only _eight_
keystrokes. Eight keystrokes that makes it obvious to anybody reading
the code that there are now two descriptors and you have to close both
the original descriptor and the socket.

When you create a Python file object from a file descriptor using
os.fdopen(), does it dup the descriptor? No. Would a reasonable
person expect socket.fromfd() to duplicate the descriptor? No.

Should it?

No.

I know... that particular mistake is set in stone now, and it's not
going to change. But I feel better. :)


$ python
Python 2.7.12 (default, Dec 6 2016, 23:41:51)
[GCC 4.9.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import this
The Zen of Python, by Tim Peters

Beautiful is better than ugly.
**** Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
**** Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
**** In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

</rant>

[Christian Heimes]

I wanted to fix the function before 3.6.0 came out but I faced some
resistance. My approach was deemed too magic and not fully functional on
some platforms. Other core devs are underestimating the severity of the
issue. Even in simple examples with IPv4 and IPv6 it breaks
getpeername(). I'd appreciate if you could jump in a leave a comment on
the ticket to show your support. :)

By the way I just pushed another commit with a new feature just for you.
But see for yourself:

https://github.com/tiran/socketfromfd/commit/0a2fd6dae86267cedea5ae8b956e2876e6057c74

Christian

[Antoon Pardon]

Op 22-01-17 om 01:52 schreef Grant Edwards:

> Newsgroups: gmane.comp.python.general
> From: Grant Edwards <grant.b...@gmail.com>
> Subject: Re: How to create a socket.socket() object from a socket fd?
> References: <o60naq$bbm$1...@blaine.gmane.org> <o60o2r$6cd$1...@blaine.gmane.org> <c7e3116f-6e9c-5343...@python.org>
> Followup-To:
>
> <rant>
>
> I'm still baffled why the standard library fromfd() code dup()s the
> descriptor.
>
> According to the comment in the CPython sources, the author of
> fromfd() is guessing that the user wants to be able to close the
> descriptor separately from the socket.
>
> If the user wanted the socket object to use a duplicate descriptor for
> some reason, the caller should call os.dup() -- it's only _eight_
> keystrokes. Eight keystrokes that makes it obvious to anybody reading
> the code that there are now two descriptors and you have to close both
> the original descriptor and the socket.
>
> When you create a Python file object from a file descriptor using
> os.fdopen(), does it dup the descriptor? No. Would a reasonable
> person expect socket.fromfd() to duplicate the descriptor? No.
>
> Should it?
>
> No.

The standard response to issues like this is:

A foolish consistency is the hobgoblin of little minds

--
Antoon Pardon

[Grant Edwards]

And wise consistency is the foundation of a good language design.

--
Grant Edwards grant.b.edwards Yow! I'm wet! I'm wild!
at
gmail.com

[Ethan Furman]

On 01/23/2017 09:00 AM, Grant Edwards wrote:

> On 2017-01-23, Antoon Pardon wrote:

>> The standard response to issues like this is:
>>
>> A foolish consistency is the hobgoblin of little minds
>
> And wise consistency is the foundation of a good language design.

Otherwise known as: if there's not a /very/ good reason to make them different, make them the same.

--
~Ethan~