implications of HTTP 408

[Troy Cauble]

I've got a gevent+requests HTTP client that's blasting a RESTful service at high speed with 50 threads.  (pool.spawn)

On rare occasions I get a 408 returned, which means the server thinks the client took too long to write the request.

I imagine that my the client thread, instead of blocking on open() gives up control,

then doesn't get control back in time to do the write.

Can I assume that there's an order to threads getting a turn?  I think so.

So, assuming the TCP finished and my thread should've seen open() complete,

this means that 49 threads didn't give up control in that time?

Does this theory seem right?

Any ideas for debugging or fixing?

-troy

[Jason Madden]

> On Nov 17, 2016, at 15:57, Troy Cauble <troyc...@gmail.com> wrote:
>
> I've got a gevent+requests HTTP client that's blasting a RESTful service at high speed with 50 threads. (pool.spawn)

I assume you mean greenlets? gevent.pool.Pool? Threads would be a different matter.

> On rare occasions I get a 408 returned, which means the server thinks the client took too long to write the request.
>
> I imagine that my the client thread, instead of blocking on open() gives up control,
> then doesn't get control back in time to do the write.
>
> Can I assume that there's an order to threads getting a turn? I think so.

There is an order, but it's not necessarily deterministic once there's IO involved.

> So, assuming the TCP finished and my thread should've seen open() complete,
> this means that 49 threads didn't give up control in that time?
>
> Does this theory seem right?

That seems reasonable.

> Any ideas for debugging or fixing?

You might try using one of the greenlet switch tracers you can find by googling to verify that none of your greenlets are doing some non-cooperative task and blocking other greenlets from switching in...or perhaps are dealing with a substantially bigger request/response than the others.