Setting order/priority to callbacks

[Ashish]

Hi,

I've these callbacks in my server code that runs on Windows:

on_read: after reading from socket
after_worker_thread: after worker thread ends
on_keystroke: after user hits keystroke

However, sometimes event loop continuously keep invoking on_read only and no other callbacks will be invoked for long time
(e.g. even if user presses keys or worker thread has ended long time back etc.)

This delays critical tasks and results in bad performance, resource consumption etc.

Is there any way to address this situation? (e.g. Setting order/priority to callbacks etc.?)

Tnx,
Ashish

[Saúl Ibarra Corretgé]

Hi,

On 1/5/14 2:46 PM, Ashish wrote:
>
> Hi,
>
> I've these callbacks in my server code that runs on Windows:
>

> *on_read:* after reading from socket

I assume this is the read callback passed to uv_read_start.

> *after_worker_thread: *after worker thread ends

Do you mean the after_work_cb passed to uv_queue_work?

> *on_keystroke:* after user hits keystroke

Libuv doesn't have such a callback, what is your application doing to
fire it?

>
> However, sometimes event loop continuously keep invoking on_read only
> and no other callbacks will be invoked for long time
> (e.g. even if user presses keys or worker thread has ended long time
> back etc.)
>
> This delays critical tasks and results in bad performance, resource
> consumption etc.
>
> Is there any way to address this situation? (e.g. Setting order/priority
> to callbacks etc.?)
>

There is no way to set priorities to callbacks, they are always fired in
a consistent order: timers, idle, prepare, block for i/o, check, ...

It's hard to tell what's happening because I know nothing about what
your application is doing. Do you have a libuv-only test case which
behaves incorrectly?


Cheers,

--
Sa�l Ibarra Corretg�
http://bettercallsaghul.com

[Ashish]

Sa�l Ibarra Corretg�
http://bettercallsaghul.com

Yes, on_read is the read callback passed to uv_read_start and after_work_cb is passed to uv_queue_work.

on_keystroke is callback passed to uv_read again but for tty stream.

Sorry I do not have libuv only code at the moment, but it is simple tcp server on Windows. 

It reads input data (predefined size) from socket and to process that data it fires fixed number of threads using uv_queue_work.

Now when I connect to this server using simulator (which simulates thousand connections) and send loads of *asynchronous* data to server, 

sometimes only on_read gets fired continuously for long time despite worker threads are done with processing for quite a time back, 

I do not get after_work_cb for any of them, neither do I get tty callback even after keyboard input.

 

[Saúl Ibarra Corretgé]

>
> Yes, on_read is the read callback passed to uv_read_start and
> after_work_cb is passed to uv_queue_work.
> on_keystroke is callback passed to uv_read again but for tty stream.
>

I see.

> Sorry I do not have libuv only code at the moment, but it is simple tcp
> server on Windows.
> It reads input data (predefined size) from socket and to process that
> data it fires fixed number of threads using uv_queue_work.
>

uv_queue_work is an opaque thread pool, what do you mean by "fixed
number of threads"? On Windows it uses QueueUserWorkItem
http://msdn.microsoft.com/en-us/library/windows/desktop/ms684957(v=vs.85).aspx
so I guess it could take a while to execute if you saturate it. But I'm
just guessing here.

Side question: why do you process the data in a different thread? What
kind of processing are you doing?

> Now when I connect to this server using simulator (which simulates
> thousand connections) and send loads of *asynchronous* data to server,
> sometimes only on_read gets fired continuously for long time despite
> worker threads are done with processing for quite a time back,
> I do not get after_work_cb for any of them, neither do I get tty
> callback even after keyboard input.
>

As for not getting the TTY callbacks, I don't really know why, sorry.


Regards,

[Ashish]

Sa�l Ibarra Corretg�
http://bettercallsaghul.com

Hi Saúl,

Thanks really for your looking into this.

If we call uv_queue_work unconditionally it goes on creating thousands of threads (I did see it through Windows task manager). It was becoming tough to manage resources in that case. To avoid that, I call uv_work_queue only for predefined number of times, and next call(s) will occur only after I got callback(s) to previous call(s).

My event loop puts incoming data (requests) in a queue and routine passed to uv_work_queue (thread) processes those requests in queue sequentially. Thus, event loops job is only to dump incoming requests to queue through on_read callback, and to attend other callbacks (after_work_cb, tty callback etc)

Currently it is simple echo server. So nothing much I am doing in threads except copy the request (for sake of doing something), so that after_work_cb can send it back to client.

It all works fine. But when I simulate thousands of connections those bombards my server with continuous asynchronous requests, event loop keeps invoking only on_read callback. Thus, after_work_cb never gets called and my request queue goes on increasing forever to system memory limits. (TTY callback also never gets called but I guess underlying reason might be the same for both)

What I observed additionally is, once I stop simulator, event loop immediately resumes calling back these other callback.

Tnx,

Ashish

[Saúl Ibarra Corretgé]

Hi,

>
> Hi Sa�l,

>
> Thanks really for your looking into this.
>
> If we call uv_queue_work unconditionally it goes on creating thousands
> of threads (I did see it through Windows task manager). It was becoming
> tough to manage resources in that case. To avoid that, I call
> uv_work_queue only for predefined number of times, and next call(s) will
> occur only after I got callback(s) to previous call(s).
>

I see.

> My event loop puts incoming data (requests) in a queue and routine
> passed to uv_work_queue (thread) processes those requests in queue
> sequentially. Thus, event loops job is only to dump incoming requests to
> queue through on_read callback, and to attend other callbacks
> (after_work_cb, tty callback etc)
>
> Currently it is simple echo server. So nothing much I am doing in
> threads except copy the request (for sake of doing something), so that
> after_work_cb can send it back to client.
>
> It all works fine. But when I simulate thousands of connections those
> bombards my server with continuous asynchronous requests, event loop
> keeps invoking only on_read callback. Thus, after_work_cb never gets
> called and my request queue goes on increasing forever to system memory
> limits. (TTY callback also never gets called but I guess underlying
> reason might be the same for both)
>
> What I observed additionally is, once I stop simulator, event loop
> immediately resumes calling back these other callback.
>

Since what you are implementing is an echo server, thus i/o bound, lets
try to create a simpler test case: don't use uv_queue_work at all. Just
do the uv_write on the read callback to send the reply back.

Then hammer your server with those thousands of connections and lets see
how that goes.

Regards,

--
Sa�l Ibarra Corretg�

bettercallsaghul.com

[Ashish]

On Tuesday, January 7, 2014 9:08:10 PM UTC+5:30, Saúl Ibarra Corretgé wrote:

Hi,

>
> Hi Sa�l,

Sa�l Ibarra Corretg�
bettercallsaghul.com

Hi  Saúl,

I tried as you suggested by not using uv_work_queue and doing just uv_write on the read callback and then hammering with thousand connections and requests.
It works absolutely perfect and I also get other callbacks (after_write, TTY callback)

( However, the echo server is just for test purpose. In production scenario I have to have worker threads to do CPU bound processing on all incoming requests before I send results to client(s) )

Now I still didn't get any clue,   with massive incoming requests from thousands of clients, after doing uv_work_queue calls why only read callback works and other are inactive?

Many Thanks,
Ashish

[Saúl Ibarra Corretgé]

>
> Hi Sa�l,

>
> I tried as you suggested by not using uv_work_queue and doing just
> uv_write on the read callback and then hammering with thousand
> connections and requests.
> It works absolutely perfect and I also get other callbacks (after_write,
> TTY callback)
>

Ok, so the problem seems to come from the threadpool thing then.

> ( However, the echo server is just for test purpose. In production
> scenario I have to have worker threads to do CPU bound processing on all
> incoming requests before I send results to client(s) )
>

I see.

> Now I still didn't get any clue, with massive incoming requests from
> thousands of clients, after doing uv_work_queue calls why only read
> callback works and other are inactive?
>

Looks like the after work callbacks are not getting called for some
reason. I'm no windows expert, and I don't see any obvious bug in the
windows code, so I really have no clue here :-S

Maybe using the same threadpool for Windows (using the one we have for
Unix, that is) would be better. Have a look at this issue comment:
https://github.com/joyent/libuv/issues/649#issuecomment-14726848

It shouldn't be difficult to port the Unix threadpool to Windows, but
AFAIK no one is working on it at the moment.


Cheers,

[Ashish]

On Wednesday, January 8, 2014 1:15:28 AM UTC+5:30, Saúl Ibarra Corretgé wrote:

>
> Hi  Sa�l,

Sa�l Ibarra Corretg�
bettercallsaghul.com

Hi Saúl,

It seems there is inconsistency involved in here. I repeated your told "uv_write on the read callback itself without having uv_work_queue"
experiment again today for multiple times in sequence. And I found that the execution was not consistent each time.
Sometimes it is perfect but sometimes (rather abruptly) after_write callbacks suddenly starts getting too much delayed. (Same is for TTY callback)
That delay causes not freeing up allocations, resulting in huge memory hog. (Note: uv_write does never returned any error)

What I observed additionally is that when the things are in perfect order, CPU utilization of System (OS kernel) and server (my code+libuv) are 50% 50% each.
But when it starts delaying after_write callback, CPU for System goes down to 7 percent while server is still at full throttle.

Now what could be reasons for this abrupt behavior. (By the way, I run  simulator and server on same machine. But I hope that shouldn't contribute to this?)

Tnx,
Ashish