self.flush() is too fast therefore blocks the server

[bhch]

I've been doing some performance testing of Tornado while serving large files.

Consider this code.

class MyHandler(...):
    async def get(self):
        while True:
            # read a chunk
            self.write(chunk)
            await self.flush()

Since, `self.flush()` returns a Future, we can `await` on it while it writes the 

chunk to socket. This means that while the data is being written to socket, our 

server is free to serve other clients. 

But that's not always the case. Most of the times, especially if the chunk is small 

< 5 MiB, `self.flush()` is very fast, so, the current handler doesn't suspend at 

`await`, and therefore, it keeps serving the file. 

For larger chunks, sometimes it takes a little longer to write to the socket, and then 

the server will serve other clients, but it's still not guaranteed.

As a workaround, I put the handler to sleep for a nanosecond right after flush, so 

that the server doesn't block and can serve other clients. And this works.

However, I'd like to know 

1. if this approach is good or not? 
2. if there's a better way to pause the server at `self.flush()`? 

[Ben Darnell]

Yes, this is good if you find yourself sending large amounts of data over a fast network. You may also want to consider making the "read a chunk" step asynchronous if that would give the system another chance to serve another coroutine (if you're reading the file from disk, consider using a thread pool). 

Incidentally, Tornado 5.0 will be much more efficient at writing large amounts of data like this.

 

1. if there's a better way to pause the server at `self.flush()`? 

To wait for the shortest possible time (one IOLoop iteration), you can use `await None` instead of sleeping for a nanosecond. These are equivalent since an IOLoop iteration takes many nanoseconds, but `await None` is a little more efficient and easier to type. 

Alternately, depending on your application, you may want to serve each client at a "fair" rate instead of allowing a single client to consume as much bandwidth as you can give it. In this case you'd want to sleep for something longer than a nanosecond, or use a metering pattern like this:

    while True:

        # Start the clock to ensure a steady maximum rate

        deadline = IOLoop.current().time() + 0.1

        # Read a 1MB chunk

        self.write(chunk)

        await self.flush()

        # This sleep will be instant if the deadline has already passed;

        # otherwise we'll sleep long enough to keep the transfer

        # rate around 10MB/sec (adjust the numbers above as needed

        # for your desired transfer rate)

        await gen.sleep(deadline)

-Ben

 

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[bhch]

The "metering pattern" is a really creative idea. I also gave Tornado 5.0 a try and I found it to be 

around 18-20% faster that v4.5. 

One thing I'd like to point out is that `await None` doesn't work as it raises an exception that says 

"NoneType can't be used in await". However, `yield None` does work if used with `@gen.coroutine` 

instead of `async def`. 

Many thanks for an insightful reply.

[Ben Darnell]

Ah, you're right. `await None` doesn't work, and the recommended replacement (in asyncio) is `await asyncio.sleep(0)` (https://github.com/python/asyncio/issues/284). So I guess a short sleep would be the best thing to do in Tornado as well (in Tornado 5.0, `await asyncio.sleep(0)` will be faster than using `tornado.gen.sleep` in an `async def` coroutine). Maybe I should bring back `gen.moment` as a non-None object that can be used for the same purpose. `asyncio.Future` makes this tricky, though. 

-Ben

--