EM::defer improves performance?

[Arpan]

Hi, 

I have to process high volume data coming off the server side onto the client side. I am trying to understand what approach does EM recommend - create a EM.Queue and push the items onto the queue and have a separate Ruby thread to read data off EM.Queue OR use EM::Defer and schedule a callback to read data off the EM.Queue? 

I think the latter is preferable since EM::Defer may create (possibly) multiple threads and processing becomes a lot faster. Please advise.

Arpan

[James Tucker]

On MRI, threads are not parallel, so CPU intensive jobs will not get faster as a result of adding threads.

JRuby and Rubinius have real threads.

If you need parallelism on MRI, then use a load balancer in front of multiple processes.

 

Arpan

--
You received this message because you are subscribed to the Google Groups "EventMachine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eventmachine...@googlegroups.com.
To post to this group, send email to eventm...@googlegroups.com.
Visit this group at http://groups.google.com/group/eventmachine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[hemant]

If the Data processing is CPU intensive using EM.defer may not help, because of VM lock and in fact data processing thread may starve out the thread which processes Events. Also, EM.defer is backed by a thread pool and was designed to be used for stuff that requires blocking IO (a protocol which can't be multiplexed easily). 

It all depends on - volume of Data and what kind of processing you are doing with it.

If possible, I would recommend looking into one process for handling incoming data and another process for processing it.  What I would do is -

1. Master process that handles socket stuff (using EM) and it writes the read data to a message queue (I am thinking simple POSIX Message Queues should suffice (https://github.com/tmm1/ruby_posix_mq).

2. Another process reads the data from POSIX message queue and does the required processing.

Arpan

--

You received this message because you are subscribed to the Google Groups "EventMachine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eventmachine...@googlegroups.com.
To post to this group, send email to eventm...@googlegroups.com.
Visit this group at http://groups.google.com/group/eventmachine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Let them talk of their oriental summer climes of everlasting conservatories; give me the privilege of making my own summer with my own coals.

http://gnufied.org
http://twitter.com/gnufied