tornado.iostream udp

[Imran Farooq]

Hi,

I am writing a tornado server application and for my application to work i need to send asynch requests to other services (listening on different udp ports). I transfer pickled dictionary objects between tornado server app and my udp services. Normally, every thing works fine except when the packet size becomes larger than 4000 bytes. When the size becomes greater than 4000 bytes the packets at tornado callback end (set by tornado.iostream.ready_bytes) gets corrupted. so if my packet structure is

4150\naljasdfsbasdfsd....................................................................4000lj;lja;lsjdf;asjd;fjkasd;fja;s\n\n.

the packet when received becomes 

4150\naljasdfsbasdfsd....................................................................40004150\naljasdfsbasdfsd.........

the received packet has exactly the same no of bytes but the packets is wrapped around from the end maybe after a fixed no. of bytes  (I haven't been able to find out the exact size after which the problem occurs but all my packets which are above 4000 bytes get corrupted). For the time being both my tornado server and the udp service i am communicating with, run on same machine on different ports. Also to be sure I have checked packed delivered on these ports using wireshark, and it seems that packets received at tornado side are perfectly fine with no corruption. But still in my callback I get a corrupted packet or you can say incoplete packet with last few bytes wrapped around.

Is there a fix buffer size in tornado.iostream library. Can we send large udp packets like these using tornado.iostream ???

Please help

Regards,
Imran

[Ben Darnell]

IOStream doesn't really make sense for UDP traffic, since packets can
be delivered out of order or not at all. You'll probably need a class
that's like IOStream but just reads a packet at a time instead of
methods like read_bytes/read_until.

Note that large UDP packets can be unreliable in general, unless you
can make assumptions about the network between client and server.
Many UDP protocols limit themselves to 512-byte packets for this
reason. http://stackoverflow.com/questions/1098897/what-is-the-largest-safe-udp-packet-size-on-the-internet

Also, it's generally a bad idea to use pickled objects in a network
protocol - decoding pickled objects is essentially eval()ing arbitrary
code, so it's not safe to receive pickled objects from untrusted
sources.

-Ben

[Imran Farooq]

For the time being my tornado server and udp service both reside on same pc. And before opting for tornado I used simple udp server thread on http server side to receive messages from udp service, and it worked for about a year without trouble. Now with tornado since threading is discouraged, i started using iostream, and it almost works in all cases accept when the packet size is too large (same packet size used to work fine form python udp socket server earlier).

Are there iostream type libraries which provide callback based mechanism. I cannot use the polling method to see if udp side has sent some data. My server serves a real time application and I am using comet programming to do this. Any updates from udp side must be sent back to client in real time and to do this I need to raise an event on tornado web server side.

Any solutions.

[Ben Darnell]

On Fri, Nov 19, 2010 at 9:42 AM, Imran Farooq
<shaikhimr...@gmail.com> wrote:
> For the time being my tornado server and udp service both reside on same pc.
> And before opting for tornado I used simple udp server thread on http server
> side to receive messages from udp service, and it worked for about a year
> without trouble. Now with tornado since threading is discouraged, i started
> using iostream, and it almost works in all cases accept when the packet size
> is too large (same packet size used to work fine form python udp socket
> server earlier).
>

I wouldn't say threading is discouraged with tornado. If you've got
something that works in a separate thread, there's nothing wrong with
continuing to use that with tornado. Just remember to use
IOLoop.instance().add_callback() for the handoff from the UDP server
thread to the tornado thread.

> Are there iostream type libraries which provide callback based mechanism. I
> cannot use the polling method to see if udp side has sent some data. My
> server serves a real time application and I am using comet programming to do
> this. Any updates from udp side must be sent back to client in real time and
> to do this I need to raise an event on tornado web server side.
>

I don't know of any IOStream-like UDP modules for tornado, but it
shouldn't be too hard to write one. You may not even need much of a
library since there's no buffering (just use IOLoop.add_handler
directly), but given the problems you're seeing with IOStream there
may be some tricks around reassembling fragmented packets.

-Ben

[Imran Farooq]

Thanks Ben, i will try that.

 

Do you know about any example or sample code using ioloop.instance.add_callback() thing with threads.

[Ben Darnell]

See this thread from last week:
http://groups.google.com/group/python-tornado/msg/abf55156e66b6980

-Ben

On Sat, Nov 20, 2010 at 12:26 AM, Imran Farooq