Using ernie as a possible replacement for gearman.
tim
for gearman. Currently gearman gives me the following features and I
am curious to see if bert can do the same types of things.
1. Multiple gearman servers for failover. All your "clients" (berts)
and workers (ernies) connect to all your gearman servers so if one
server fails the others can distribute the jobs. I suppose in ernie
you just use a TCP load balancer or something like that right?
2) Persistent queue. If a gearman server falls down it looks at the
backlog queue and redistribute the work.
3) Load balancing. If one worker is busy gearman finds the next
available worker to do the job. From what I can gather by reading the
docs this is not a problem with ernie.
One other question.
In gearman I normally have one daemon per task and I launch some
daemons multiple times. The example I looked at shows one ruby file
which handles multiple types of work. Is there any advantage or
disadvantage for this kind of approach? My current issue is the
memory usage by the 40+ workers I have running. These workers take up
a couple of gigs of RAM right now.
Jared "Lucky" Kuolt
Resque[1] (another a Githubber's project). It's a queue-based worker
system that seems to map to the Gearman idea more so than BERT-RPC.
Which is not to say you couldn't do it with Bert, you'd just need to
add a bit more logic in your bert-rpc system to make it fit your
requirements.
[1]: http://github.com/defunkt/resque
--
Lucky ♣ http://luckythetourist.com
Tim Uckun
<luckyth...@gmail.com> wrote:
> To be honest, it sounds like you're looking for something like
> Resque[1] (another a Githubber's project). It's a queue-based worker
> system that seems to map to the Gearman idea more so than BERT-RPC.
> Which is not to say you couldn't do it with Bert, you'd just need to
> add a bit more logic in your bert-rpc system to make it fit your
> requirements.
I looked at resque. From what I can gather it does not support
multiple resque servers which in my opinion puts it at a disadvantage
when compared with gearman, rabbitmq etc.
I should say that gearman does offer more options than most other
queue bases servers. You can choose to run a task syncronously if you
want. In other words you send a request, a worker picks it up, runs
the task and gives you back the results. The worker can give you back
updates such as "10 percent done" and worker exceptions are propagated
back. Gearman takes a lot of ram if you have a large queue or send
large amounts of data though because it's keeping all of it in RAM.
Chris Wanstrath
On Sep 17, 2010, at 10:16 AM, "Jared \"Lucky\" Kuolt" <luckyth...@gmail.com> wrote:
> To be honest, it sounds like you're looking for something like
> Resque[1] (another a Githubber's project). It's a queue-based worker
> system that seems to map to the Gearman idea more so than BERT-RPC.
> Which is not to say you couldn't do it with Bert, you'd just need to
> add a bit more logic in your bert-rpc system to make it fit your
> requirements.
>
> [1]: http://github.com/defunkt/resque
Resque is way better than Bert.
>