EventMachine-based task broker

Skip to first unread message

Arindam Mukherjee

Jul 22, 2016, 2:10:41 AM7/22/16
to EventMachine
I am trying to build a task broker using EventMachine, thread pools and a fibers. I wanted to validate the approach and ask questions about some challenges I am facing.

The task broker runs a thread-pool (perhaps not EM.defer) to which individual tasks are assigned.
The tasks are I/O bound, but occasionally may do CPU bound activity.

Proposed design:
1. EventMachine event loop runs in its own thread.
2. Each task is represented by a Ruby Proc / lambda. Worker threads from a thread pool service each task.
3. A worker thread runs the task in a fiber. There is a global EM::Queue for shipping I/O work to the EM loop. Each task creates a channel that the EM loop uses to ship data back to the task. 4. The task subscribes to the channel it creates. It does a Fiber.yield after shipping I/O to the EM loop. And in the subscription callback, it resumes itself.

This way, using Fibers to represent tasks, I hoped to have single threads multiplexing multiple tasks. But I am somehow not able to pull this off. Is this a viable approach?


James Tucker

Jul 22, 2016, 2:21:48 AM7/22/16
to eventm...@googlegroups.com

Why not just use Ruby's Queue class, and do io in your worker threads, sizing your pool appropriately to fill cpu while waiting on io?

You received this message because you are subscribed to the Google Groups "EventMachine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eventmachine...@googlegroups.com.
To post to this group, send email to eventm...@googlegroups.com.
Visit this group at https://groups.google.com/group/eventmachine.
For more options, visit https://groups.google.com/d/optout.

Arindam Mukherjee

Jul 22, 2016, 2:35:53 AM7/22/16
to EventMachine
You mean doing blocking I/O in the worker threads? Or using EventMachine somehow? Sorry for not understanding.

James Tucker

Jul 22, 2016, 2:44:38 AM7/22/16
to eventm...@googlegroups.com

Yes, I mean blocking io.

Reply all
Reply to author
0 new messages