gearman vs. rabbitmq

[bruce]

Thoughts/comments on which one is "better"

use cases, personal thoughts, etc...

not a flame war... but trying to get some insight from people who may have used both...

thanks

[Clint Byrum]

These are the differences that have affected my choice in the past:

1) RabbitMQ has obvious broadcast channels.
You can send 1 message that reaches all subscribers in RabbitMQ. In
Gearman, this can be done, but not obviously, as it requires workers to
act as clients, and broadcasting clients to, subsequently, act as
workers and use the coalescing feature of Gearman.

2) Gearman's protocol is wicked efficient for "fan out".
If you want to take tiny bits of work, and distribute it amongst a
group of workers, especially if latency matters to you, Gearman is
extremely efficient at that.

3) Gearman has coalescing built in.
This is where you can just add a unique ID to a job, and if another
client submits a job with the same unique ID, you will both get the
result of a single running of that job multi-plexed to you. This is
really useful for mitigating thundering herd problems, so that only one
work unit per unique thing is pointed at your limited resource.

Another thing RabbitMQ has that gearmand does not is strong
authentication and SSL support, but I have not used this much and so I
can't comment much on it.

[bruce]

Hi Clint.

Thanks for the reply. In my situation, I'm trying to understand the ability to "authenticate" or ensure that only valid clients can connect back to the gearman server for accessing working packets.

I'm envisioning an architecture, for a web crawling app.

The crawler would have a "fetch" and a "parse" side, with each side being implemented by a number of client servers. So, there will be fetch servers, getting the "fetch" packets from the fetch gearman server. There will also be parse servers, getting the "parse" packets/data from the parse gearman server.

As I understand gearman, there's no built in auth process to validate/ensure a client accessing the queue is able to access the data. I know I can implement a process where I target a given packet to a given clientserver, but if I want to simply do a kind of fifo, where the packets on the queue are handled by the 1st free client among the group of client "fetch" servers, then I might be out of luck. Do you know of a soln for this?

In my mind, I envision 200-300 client fetch edge/servers, sitting out on the 'net, kind of like a small seti. A distributed client/server crawler, using gearman to handle all of the packets that need to be fetched, as well as the returned packets from the fetch process, to be parsed, and in turn generates the next level of the "fetch" data...

Thanks!

--
You received this message because you are subscribed to the Google Groups "Gearman" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gearman+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Shane Harter]

You can always implement that security via iptables. 

Honestly I've never missed authentication pieces in Gearman.

To unsubscribe from this group and stop receiving emails from it, send an email to gearman+u...@googlegroups.com.

[Ankur Gupta]

Another way to implement security over the wire is to encrypt and
decrypt data at worker and jobserver level in application code.

>>> email to gearman+u...@googlegroups.com.
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Gearman" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to gearman+u...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Gearman" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to gearman+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

--
Homepage -> http://uptosomething.in

[Clint Byrum]

(just a side note, please try not to "top post" as it really screws up
the flow of conversations while composing a reply).

On 2013-05-08 07:45, Ankur Gupta wrote:
> Another way to implement security over the wire is to encrypt and
> decrypt data at worker and jobserver level in application code.
>

This helps for end-to-end auth/privacy, but you can't prevent a
malicious worker from connecting and simply consuming all of the jobs
with that scheme.

> On Wed, May 8, 2013 at 7:57 PM, Shane Harter <shane....@gmail.com>
> wrote:
>> You can always implement that security via iptables.
>>
>> Honestly I've never missed authentication pieces in Gearman.
>>
>>

Filtering by network address should work fine, its true.

I've been hoping that TLS support would land in gearmand for a while.
Brian Aker was working on it a few months ago in a branch, I even tested
it a little. Not sure where that went. Using TLS would really be the
only way to run a public facing gearmand.

You can fake that with stunnel, but its not built into any of the
client libs, so it becomes a huge hassle.

[Ankur Gupta]

On Wed, May 8, 2013 at 8:22 PM, Clint Byrum <cl...@fewbar.com> wrote:
> (just a side note, please try not to "top post" as it really screws up the
> flow of conversations while composing a reply).
>
>
> On 2013-05-08 07:45, Ankur Gupta wrote:
>>
>> Another way to implement security over the wire is to encrypt and
>> decrypt data at worker and jobserver level in application code.
>>
>
> This helps for end-to-end auth/privacy, but you can't prevent a malicious
> worker from connecting and simply consuming all of the jobs with that
> scheme.

Yes you are right, just that when the Security Admin comes to do an
audit some security is better then none. Since we run behind a
firewall we get away with this makeshift arrangement. Gearmand with IP
accessible thru Internet is a strict no no.

>
>
>> On Wed, May 8, 2013 at 7:57 PM, Shane Harter <shane....@gmail.com>
>> wrote:
>>>
>>> You can always implement that security via iptables.
>>>
>>> Honestly I've never missed authentication pieces in Gearman.
>>>
>>>
>
> Filtering by network address should work fine, its true.
>
> I've been hoping that TLS support would land in gearmand for a while. Brian
> Aker was working on it a few months ago in a branch, I even tested it a
> little. Not sure where that went. Using TLS would really be the only way to
> run a public facing gearmand.
>
> You can fake that with stunnel, but its not built into any of the client
> libs, so it becomes a huge hassle.
>
>

[Brian Aker]

Hi,

On May 7, 2013, at 9:11 PM, Clint Byrum <cl...@fewbar.com> wrote:

Another thing RabbitMQ has that gearmand does not is strong authentication and SSL support, but I have not used this much and so I can't comment much on it.

I thought you were going to finish part of the patch for this :)

I'll try to merge it again...

Cheers,

-Brian

[bruce]

Hey,

So, my asking the initial question, might actually have a patch for "auth" for gearman implemented, sweet!!

My issue, at least in my potential app/arch, is that I have a master gearman server running behind the firewall.

However, the gearman clients, which will do all the work, are all the same, and run on machines sitting on the public internet (think something like seti). In this case, I'm trying to figure out a way to auth the client app that connects back with the master gearman server to process the next work packet from the server.

I'm inclined to think I can setup a webserver/service kind of process, where I can then have the client connect with the webservice to process the gearman packet.. This adds a layer on top of the client, as I wouldn't just have the gearman client communicating back with the master gearman daemon/server.

Basically, the issue is how would one create a gearmand server behind a firewall, to allow authenticated clients on the internet to access the server, and restrict the access to only the authenticated clients..

Thoughts/Comments...

Thanks

-bruce

--

[Clint Byrum]

<ahem> please.. the top posting.. just move your cursor down below the
statements you're replying to. :)

On 2013-05-09 06:05, bruce wrote:
> Hey,
>
> So, my asking the initial question, might actually have a patch for
> "auth" for gearman implemented, sweet!!
>
> My issue, at least in my potential app/arch, is that I have a master
> gearman server running behind the firewall.
>
> However, the gearman clients, which will do all the work, are all the
> same, and run on machines sitting on the public internet (think
> something like seti). In this case, I'm trying to figure out a way to
> auth the client app that connects back with the master gearman server
> to process the next work packet from the server.
>
> I'm inclined to think I can setup a webserver/service kind of
> process, where I can then have the client connect with the webservice
> to process the gearman packet.. This adds a layer on top of the
> client, as I wouldn't just have the gearman client communicating back
> with the master gearman daemon/server.
>
> Basically, the issue is how would one create a gearmand server behind
> a firewall, to allow authenticated clients on the internet to access
> the server, and restrict the access to only the authenticated
> clients..
>

I think you want a web socket API. This is going to be the most robust
way to keep your public internet workers connected to your backend
service. Then you can just use https and build in light weight
authentication schemes.

The queue of things to do can still be in gearmand hosted behind that
API server, but I would not do gearman protocol across the internet,
even with SSL. Long lived connections are just not the greatest idea and
I'm not sure libgearman or any of the other native clients are built to
handle things like their own IP changing or etc. etc.

[Brian Moon]

On 5/9/13 10:27 AM, Clint Byrum wrote:
> <ahem> please.. the top posting.. just move your cursor down below the
> statements you're replying to. :)

You clearly don't email with any business people. They break you of
caring about this very quickly.

Brian.

[Clint Byrum]

Excerpts from Brian Moon's message of 2013-05-09 10:08:04 -0700:

I agree, in a business reply-all Outlook world, there is no way to get
sane quoting.

But on mailing lists, where the user's client clearly produces properly
quoted messages, its just typically ignorance.

[Brian Aker]

Hi!

On May 8, 2013, at 7:52, Clint Byrum <cl...@fewbar.com> wrote:

> on it a few months ago in a branch, I even tested it a little. Not sure where that went. Using TLS would really be the only way to run a public facing

Why not skip TLS and just handle Auth via certificates?

I have the SSL branch compiling as of last night. I will need to see
how well it works today.

Cheers,
-Brian