EventMachine-based task broker

25 views
Skip to first unread message

Arindam Mukherjee

unread,
Jul 22, 2016, 2:10:41 AM7/22/16
to EventMachine
I am trying to build a task broker using EventMachine, thread pools and a fibers. I wanted to validate the approach and ask questions about some challenges I am facing.

The task broker runs a thread-pool (perhaps not EM.defer) to which individual tasks are assigned.
The tasks are I/O bound, but occasionally may do CPU bound activity.

Proposed design:
1. EventMachine event loop runs in its own thread.
2. Each task is represented by a Ruby Proc / lambda. Worker threads from a thread pool service each task.
3. A worker thread runs the task in a fiber. There is a global EM::Queue for shipping I/O work to the EM loop. Each task creates a channel that the EM loop uses to ship data back to the task. 4. The task subscribes to the channel it creates. It does a Fiber.yield after shipping I/O to the EM loop. And in the subscription callback, it resumes itself.

This way, using Fibers to represent tasks, I hoped to have single threads multiplexing multiple tasks. But I am somehow not able to pull this off. Is this a viable approach?

Arindam

James Tucker

unread,
Jul 22, 2016, 2:21:48 AM7/22/16
to eventm...@googlegroups.com

Why not just use Ruby's Queue class, and do io in your worker threads, sizing your pool appropriately to fill cpu while waiting on io?


--
You received this message because you are subscribed to the Google Groups "EventMachine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eventmachine...@googlegroups.com.
To post to this group, send email to eventm...@googlegroups.com.
Visit this group at https://groups.google.com/group/eventmachine.
For more options, visit https://groups.google.com/d/optout.

Arindam Mukherjee

unread,
Jul 22, 2016, 2:35:53 AM7/22/16
to EventMachine
You mean doing blocking I/O in the worker threads? Or using EventMachine somehow? Sorry for not understanding.

James Tucker

unread,
Jul 22, 2016, 2:44:38 AM7/22/16
to eventm...@googlegroups.com

Yes, I mean blocking io.

Reply all
Reply to author
Forward
0 new messages