Could GC cause periodic spikes when processing TCP messages?

[John Chow]

Right now in our production server, we have a server Ubuntu 11.04 server running EM 1.0.0.beta.4 as one of our backends. For the most part, it receives a TCP message from one client, does some slight processing in EM::Connection#receive_data, and broadcasts that same message to other clients. On average, it takes anywhere from 1-3ms to process the message, not a big deal in my eyes.

This might be a silly question, but please bear with me: every so often (maybe 1 out 5-10 messages), the processing time spikes up from 3ms to 30ms, even though it's processing the same exact message. We have very low load, so my guess is that this is due to the GC, but I'm not experienced enough to say for a certainty (this is my first production EM server out in the wild).

If anybody could shed some light for this noob on why this could be happening, that'd be great :)

John

[Iñaki Baz Castillo]

2012/5/16 John Chow <jwiz...@gmail.com>:

> This might be a silly question, but please bear with me: every so often
> (maybe 1 out 5-10 messages), the processing time spikes up from 3ms to 30ms,
> even though it's processing the same exact message. We have very low load,
> so my guess is that this is due to the GC,

No, the TCP EM::Connection instance gets stored within a Ruby hash
until it's terminated, so it cannot be GC'd.

--
Iñaki Baz Castillo
<i...@aliax.net>

[John Chow]

Thanks for the reply Iñaki. My question wasn't about how MRI gc's the EM objects, but rather how the MRI GC affects the performance of an EM app. Since this is my first real production server, I'm not sure why some processing are 3ms and some are 30ms, even though they are processing the same message and behaving the same. I'm wondering if this kind of spike is normal, and if not, what possible (obvious) reason could this spike?

John

--
You received this message because you are subscribed to the Google Groups "EventMachine" group.

To post to this group, send email to eventm...@googlegroups.com.
To unsubscribe from this group, send email to eventmachine...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/eventmachine?hl=en.

[Iñaki Baz Castillo]

2012/5/16 John Chow <jwiz...@gmail.com>:

> Thanks for the reply Iñaki. My question wasn't about how MRI gc's the EM
> objects, but rather how the MRI GC affects the performance of an EM app.
> Since this is my first real production server, I'm not sure why some
> processing are 3ms and some are 30ms, even though they are processing the
> same message and behaving the same. I'm wondering if this kind of spike is
> normal, and if not, what possible (obvious) reason could this spike?

Ok, I understand. Yes, CG runs when Ruby wants so it could delay some
random operations in your code. However is difficult to prove that
since, as said before, CG runs "whenever".

Perhaps you could add a periodic timer which runs GC.start every 5
seconds, and log it somewhere. Maybe that lets you investigate if the
delay in your operation happens while CG.start takes place.

[John Chow]

Good idea, I'll have that a try later today and reply with my findings.

[James Tucker]

On May 16, 2012, at 6:36 AM, John Chow wrote:

Right now in our production server, we have a server Ubuntu 11.04 server running EM 1.0.0.beta.4 as one of our backends. For the most part, it receives a TCP message from one client, does some slight processing in EM::Connection#receive_data, and broadcasts that same message to other clients. On average, it takes anywhere from 1-3ms to process the message, not a big deal in my eyes.

This might be a silly question, but please bear with me: every so often (maybe 1 out 5-10 messages), the processing time spikes up from 3ms to 30ms, even though it's processing the same exact message. We have very low load, so my guess is that this is due to the GC, but I'm not experienced enough to say for a certainty (this is my first production EM server out in the wild).

Yep, sounds like GC. How big is the process (RSS), because if you keep it small enough, you won't even notice GC time. Of course, 30ms isn't all that much, it's in the realm of a couple of thread scheduler mis-alignments...

If anybody could shed some light for this noob on why this could be happening, that'd be great :)

John

--
You received this message because you are subscribed to the Google Groups "EventMachine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/eventmachine/-/1NT0BYb6qrcJ.