Distributed master and worker using http

dhurandar S

unread,

Mar 16, 2020, 9:13:32 PM3/16/20

to Gearman

Hi,

I have the following scenarios.

I have agent/sensor which runs on individual laptops and machines across geographies and timezones and every so often wakes up and asks for some work from the central master hosted in the cloud (AWS) . If Master has some work it assigns the work to the agent and the agent simply executes the work.

These agents communicate with the central master via HTTPS while asking for more work, and while sending the status of the work its doing, that's also the mechanism to provide heartbeat.

Please remember these workers are always up and keep asking for new and new work from the master in the cloud to execute.

Clients typically submit work to Master on a regular basis , they have 2 modes, do the work only once when it's submitted, do the work in an ongoing fashion at regular intervals.

Can I use java based Gearman in this scenario? If gearman is not a good fit here , can anyone recommend another system i can use ??

regards,

Rahul

Hung Nguyen

unread,

Mar 17, 2020, 1:41:57 AM3/17/20

to gea...@googlegroups.com

Hello,

Yes, thats the main purpose of gearman

--

Hưng

On Mar 17, 2020, at 08:13, dhurandar S <dhuran...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "Gearman" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gearman+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gearman/df290266-0029-4b0d-b52c-97374bb39458%40googlegroups.com.

Clint Byrum

unread,

Mar 18, 2020, 9:07:40 AM3/18/20

to gea...@googlegroups.com

Quoting dhurandar S (2020-03-16 18:13:32)

> Hi,
> I have the following scenarios.
> I have agent/sensor which runs on individual laptops and machines
> across geographies and timezones and every so often wakes up and asks
> for some work from the central master hosted in the cloud (AWS) . If
> Master has some work it assigns the work to the agent and the agent
> simply executes the work.
> These agents communicate with the central master via HTTPS while asking
> for more work, and while sending the status of the work its doing,
> that's also the mechanism to provide heartbeat.
> Please remember these workers are always up and keep asking for new and
> new work from the master in the cloud to execute.
> Clients typically submit work to Master on a regular basis , they have
> 2 modes, do the work only once when it's submitted, do the work in an
> ongoing fashion at regular intervals.

Gearman may in fact be useful here. However, it may be a bit lower level
than you are expecting, and you may need to add some utility workers to
achieve your goals.

For the workers that sleep and wake and grab work, that's exactly how
Gearman works. So your workers will connect to all of the gearman
servers and tell them the functions they can do, and then as clients
submit jobs, the workers will be woken up to do the jobs.

This is all handled quite nicely in most client libraries by simply
building a worker object, adding the servers and function callbacks, and
then entering a wait loop.

For doing immediate work, this is very simple on the client side, and
you only need to decide whether you want clients to wait for the job to
be completed, or submit them as background jobs. I highly recommend the
former, as background jobs are either volatile (gearmand dies, they do
too) or expensive (gearmand can store them in a reliable place like
MySQL or Postgres).

As far as ongoing periodic jobs, you'll need something to submit those,
such as a cron job, or a resident daemon that submits them.

> Can I use java based Gearman in this scenario? If gearman is not a good
> fit here , can anyone recommend another system i can use ??

To be clear, there's a java gearman server, which I have heard only good
things about, but haven't used. And there's java gearman client
libraries. Either are fine in your scenario, with the caveat that you
will have to do some thinking about background versus foreground, and
you'll have to implement periodic job submission yourself.

> regards,
> Rahul
>
> --
> You received this message because you are subscribed to the Google
> Groups "Gearman" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to [1]gearman+u...@googlegroups.com.

> To view this discussion on the web visit

> [2]https://groups.google.com/d/msgid/gearman/df290266-0029-4b0d-b52c-97
> 374bb39458%40googlegroups.com.
>
> References
>
> 1. mailto:gearman+u...@googlegroups.com
> 2. https://groups.google.com/d/msgid/gearman/df290266-0029-4b0d...@googlegroups.com?utm_medium=email&utm_source=footer

dhurandar S

unread,

Mar 24, 2020, 2:11:34 AM3/24/20

to gea...@googlegroups.com

Thank you, Clint,

For periodic job submission, I am planning to use Scheduler as the backend.

But can Gearman Master and Gearman Worker, talk to each other over HTTP or HTTPS?

regards,

Rahul

To unsubscribe from this group and stop receiving emails from it, send an email to gearman+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gearman/158453685618.27672.12594311793528849969%40clint-ThinkPad-X1-Carbon-6th.

Clint Byrum

unread,

Apr 16, 2020, 11:10:53 AM4/16/20

to gea...@googlegroups.com

gearmand has an http protocol plugin built-in, but I've not found many
users who make use of it.

http://gearman.org/manual/job_server/ has an "HTTP" section.

Still, I've oft wondered if we could make a "v2" of gearman's protocol
which is basically just "GEARMAN over WEBSOCKETS". :)

Quoting dhurandar S (2020-03-23 23:11:20)

> > an email to [1][2]gearman+u...@googlegroups.com.

> > To view this discussion on the web visit
> >

> [2][3]https://groups.google.com/d/msgid/gearman/df290266-0029-4b0d-b
> 52c-97
> > 374bb39458%[4]40googlegroups.com.
> >
> > References
> >
> > 1. mailto:[5]gearman+u...@googlegroups.com
> > 2.
> [6]https://groups.google.com/d/msgid/gearman/df290266-0029-4b0d-b52c
> -97374b...@googlegroups.com?utm_medium=email&utm_source=footer

> --
> You received this message because you are subscribed to the Google
> Groups "Gearman" group.
> To unsubscribe from this group and stop receiving emails from it,

> send an email to [7]gearman+u...@googlegroups.com.

> To view this discussion on the web visit

> [8]https://groups.google.com/d/msgid/gearman/158453685618.27672.1259
> 4311793528849969%40clint-ThinkPad-X1-Carbon-6th.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Gearman" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to [9]gearman+u...@googlegroups.com.

> To view this discussion on the web visit

> [10]https://groups.google.com/d/msgid/gearman/CALm7p6A%2BpnnyZKoGi6NLi%
> 3D8_cme1erU6S3F-r-6%2BqDQcX6tZtg%40mail.gmail.com.
>
> References
>
> 1. mailto:gea...@googlegroups.com
> 2. mailto:gearman+u...@googlegroups.com
> 3. https://groups.google.com/d/msgid/gearman/df290266-0029-4b0d-b52c-97
> 4. http://40googlegroups.com/
> 5. mailto:gearman+u...@googlegroups.com
> 6. https://groups.google.com/d/msgid/gearman/df290266-0029-4b0d...@googlegroups.com?utm_medium=email&utm_source=footer
> 7. mailto:gearman+u...@googlegroups.com
> 8. https://groups.google.com/d/msgid/gearman/158453685618.27672.12594311793528849969@clint-ThinkPad-X1-Carbon-6th
> 9. mailto:gearman+u...@googlegroups.com
> 10. https://groups.google.com/d/msgid/gearman/CALm7p6A+pnnyZKoGi6NLi=8_cme1erU6S3F-...@mail.gmail.com?utm_medium=email&utm_source=footer

bruce

unread,

Apr 16, 2020, 11:16:10 AM4/16/20

to gea...@googlegroups.com

On Thu, Apr 16, 2020 at 11:10 AM 'Clint Byrum' via Gearman <gea...@googlegroups.com> wrote:

gearmand has an http protocol plugin built-in, but I've not found many
users who make use of it.

http://gearman.org/manual/job_server/ has an "HTTP" section.

Still, I've oft wondered if we could make a "v2" of gearman's protocol
which is basically just "GEARMAN over WEBSOCKETS". :)

Hey...

For those who aren't familiar with WEBSOCKETS, and might not get it if you hit google, could you provide a short explanation of what you're considering?!!

thanks

To unsubscribe from this group and stop receiving emails from it, send an email to gearman+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gearman/158704984911.30162.8325869207373435174%40clint-ThinkPad-X1-Carbon-6th.

Clint Byrum

unread,

Apr 16, 2020, 3:46:15 PM4/16/20

to gea...@googlegroups.com

Hi Bruce. Websockets is just a streaming protocol that rides on HTTP:

https://tools.ietf.org/html/rfc6455

Gearman needs protocols like this for some things, so it would probably
make sense to replace the rather esoteric (though simple and efficient)
native protocol with one that works using websocket and http. That would
allow users to implement workers and clients more easily without needing
to have a native library like the rustygear lib I published today. :)

Quoting bruce (2020-04-16 08:15:54)

> On Thu, Apr 16, 2020 at 11:10 AM 'Clint Byrum' via Gearman
> <[1]gea...@googlegroups.com> wrote:
>
> gearmand has an http protocol plugin built-in, but I've not found
> many
> users who make use of it.

> [2]http://gearman.org/manual/job_server/ has an "HTTP" section.

> [1][2][4]gearman+u...@googlegroups.com.

> > > To view this discussion on the web visit
> > >
> >

> [2][3][5]https://groups.google.com/d/msgid/gearman/df290266-0029-4b0
> d-b
> > 52c-97
> > > 374bb39458%[4][6]40googlegroups.com.
> > >
> > > References
> > >
> > > 1. mailto:[5][7]gearman+u...@googlegroups.com
> > > 2.
> >
> [6][8]https://groups.google.com/d/msgid/gearman/df290266-0029-4b0d-b
> 52c
> >
> -[9]97374b...@googlegroups.com?utm_medium=email&utm_source=footer

> > --
> > You received this message because you are subscribed to the
> Google
> > Groups "Gearman" group.
> > To unsubscribe from this group and stop receiving emails from
> it,

> > send an email to [7][10]gearman+u...@googlegroups.com.

> > To view this discussion on the web visit
> >

> [8][11]https://groups.google.com/d/msgid/gearman/158453685618.27672.
> 1259

> > 4311793528849969%40clint-ThinkPad-X1-Carbon-6th.
> >
> > --
> > You received this message because you are subscribed to the
> Google
> > Groups "Gearman" group.
> > To unsubscribe from this group and stop receiving emails from
> it, send

> > an email to [9][12]gearman+u...@googlegroups.com.

> > To view this discussion on the web visit
> >

> [10][13]https://groups.google.com/d/msgid/gearman/CALm7p6A%2BpnnyZKo
> Gi6NLi%
> > 3D8_cme1erU6S3F-r-6%2BqDQcX6tZtg%[14]40mail.gmail.com.
> >
> > References
> >
> > 1. mailto:[15]gea...@googlegroups.com
> > 2. mailto:[16]gearman+u...@googlegroups.com
> > 3.
> [17]https://groups.google.com/d/msgid/gearman/df290266-0029-4b0d-b52
> c-97
> > 4. [18]http://40googlegroups.com/
> > 5. mailto:[19]gearman+u...@googlegroups.com
> > 6.
> [20]https://groups.google.com/d/msgid/gearman/df290266-0029-4b0d-b52
> c-97374...@googlegroups.com?utm_medium=email&utm_source=footer
> > 7. mailto:[21]gearman+u...@googlegroups.com
> > 8.
> [22]https://groups.google.com/d/msgid/gearman/158453685618.27672.125
> 94311793528849969@clint-ThinkPad-X1-Carbon-6th
> > 9. mailto:[23]gearman+u...@googlegroups.com
> > 10.
> [24]https://groups.google.com/d/msgid/gearman/CALm7p6A+pnnyZKoGi6NLi
> =8_cme1erU6S3F-...@mail.gmail.com?utm_medium=email&utm_so
> urce=footer

> --
> You received this message because you are subscribed to the Google
> Groups "Gearman" group.
> To unsubscribe from this group and stop receiving emails from it,

> send an email to [25]gearman+u...@googlegroups.com.

> To view this discussion on the web visit

> [26]https://groups.google.com/d/msgid/gearman/158704984911.30162.832
> 5869207373435174%40clint-ThinkPad-X1-Carbon-6th.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Gearman" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to [27]gearman+u...@googlegroups.com.

> To view this discussion on the web visit

Reply all

Reply to author

Forward