asyncio, datagram transports and recvmsg

Skip to first unread message


Sep 24, 2018, 11:27:16 PM9/24/18
to python-tulip
Hello asyncio developers,

the datagram transports provided by asyncio cover only some of what a
datagram socket can do. In particular, I'm missing a way to run
`recvmsg` when something is to be read, rather than `recvfrom` which is
sufficient to satisfy the datagram transport's interface.

Right now, the only way I see to get information about a message's
destination address (RECVPKTINFO) or details of ICMP errors (RECVERR)
out of a received package is reaching into asyncio internals in a way I
should not[1], which is bound to fail sooner or later, and especially
with alternative loop implementations.

Before I rush ahead and propose a concrete extension to
DatagramProtocol, I'd like to explore the options with you. Possible
paths I see out of this are:

* Export something like loop._add_reader as part of the public
loop interface.

Such a function would allow users to implement their own transports on
anything that has a file handler than can become readable using the
given selector.

Adding this would require very little effort for loops that support
that concept (eg. gbulb), and might be impossible for others -- where
it's probably OK because the underlying platform might not even
support other required concepts either. (Case in point: Windows
doesn't implement RECVPKTINFO anyway.)

* Exporting a third line of protocols / transports

Next to having a DatagramTransport / DatagramProtocol, there could be
a DatagramRecvmsgTransport / DatagramRecvmsgProtocol with all their
construction methods of loops.

A DatagramRecvmsgProtocol would have a datagram_received(data, addr,
flags, ancdata) callback (with two additional arguments compared to
DatagramProtocol's), and an additional error_msg_received(addr, flags,
ancdata) callback where requested error information is passed along.

The transport would, in addition to .sendto(addr, data), offer a
.sendmsg(data, ancdata, flags, address) method.

This would mean considerable effort (and probably duplication) on the
side of loop implementors, but solves the issue.

* Doing something fancy to make recvmsg an optional feature of datagram

Asyncio could generally run `recvmsg` rather than `recvfrom` on
datagram sockets, fetching the error queue if so indicated by the
return values of recvmsg. Then it could send the resulting data to a
datagram_msg_received(data, addr, flags, ancdata) callback on the
DatagramProtocol, which -- unless overridden by the implementation --
could fall back to DatagramProtocol's default handler, which would
then throw away the ancdata and call the established callback of

I'm not sure whether this is possible within the API guarantees given
in asyncio (eg. it would rely on protocols to inherit
DatagramProtocol, and if someone already uses their own
datagram_msg_received method, they'd accidentally overwrite something
now) -- maybe it can be made to work.

This would mean a little less effort for third party main loops, but
would provide a neater interface IMO.

The only alternatives I see to adding any of that are doing as I do now
(hooking into the _add_reader of the individual loops) or moving
networking into a different thread that runs on a loop where I can do
that (which kind of defeats the point of asyncio).

Which of the sketched ways do you think are worth pursuing, or should
this all be handled differently?

Best regards


To use raw power is to make yourself infinitely vulnerable to greater powers.
-- Bene Gesserit axiom

Yury Selivanov

Sep 24, 2018, 11:35:54 PM9/24/18
to python-tulip, chrysn
On Sep 24, 2018, 11:27 PM -0400, chrysn <>, wrote:

* Export something like loop._add_reader as part of the public
loop interface. 

Would they work for you?  I imagine you can combine loop.add_reader() and loop.add_writer() with socket.recvmsg() pretty easily.

Adding a new kind of transports for UDP is possible but would require a pretty serious motivating use case.  



Sep 26, 2018, 4:52:33 AM9/26/18
to Yury Selivanov, python-tulip
On Mon, Sep 24, 2018 at 11:35:45PM -0400, Yury Selivanov wrote:
> We already have them as part of the public API:
> Would they work for you?  I imagine you can combine loop.add_reader() and loop.add_writer() with socket.recvmsg() pretty easily.

I don't know how I overlooked that -- thanks for pointing it out this
interface, it does allow me to build all I need, and things work on
uvloop just as well as they do on asyncio.

For reference, if anyone needs a RecvmsgDatagramTransport /
RecvmsgDatagramProtocol pair, mine is implemented at [1].

Thank you for you quick and helpful response


There's always a bigger fish.
-- Qui-Gon Jinn
Reply all
Reply to author
0 new messages