Announcing Rustygear!
150 views
Skip to first unread message
Clint Byrum
Apr 16, 2020, 11:20:47 AM4/16/20
to gea...@googlegroups.com
Greetings Gearman enthusiasts and users!
Over the past few years, I've been quite enamored with Rust[1][2].
As a result, I couldn't just sit by idly while gearmand languishes in
C/C++ antiquity.
So, in my not-quite-ample spare time, I've been writing both a server
and client in Rust.
With some extra time on my hands since being laid off last month[3], I was
able to get them to a pretty usable spot.
So, without furter ado, I give you: Rustygear, and Rustygeard
https://crates.io/crates/rustygear
https://crates.io/crates/rustygeard
The former is a VERY basic client that supports only submitting normal
jobs and doesn't quite have the more modern bits of the protocol like
GRAB_JOB_UNIQ plumbed in yet. A big hole that should be fairly easy to
fill is just making it work with multiple gearman servers (right now it
allows adding many, but only works with the first one added).
However, I've tested them with both gearmand and libgearman, and
everything seems to function at the surface level.
If you want to play, you'll need to build from source, and thus, get a
rusty nightly version up and running (actually it might run on stable
now.. would love to hear if you can get it to).
To the other gearmand maintainers: How would you feel about deprecating
the C++ bits and focusing on this one? Could you make it work for your
org? What features would you need? I *really* don't want to work in
C/C++ anymore. :)
Anyway, thanks for reading, and playing. Cheers!
[1] http://fewbar.com/2017/01/a-love-letter-to-rust/
[2] http://fewbar.com/rust-for-justice/
[3] https://twitter.com/spamaps/status/1245012599873261575
Over the past few years, I've been quite enamored with Rust[1][2].
As a result, I couldn't just sit by idly while gearmand languishes in
C/C++ antiquity.
So, in my not-quite-ample spare time, I've been writing both a server
and client in Rust.
With some extra time on my hands since being laid off last month[3], I was
able to get them to a pretty usable spot.
So, without furter ado, I give you: Rustygear, and Rustygeard
https://crates.io/crates/rustygear
https://crates.io/crates/rustygeard
The former is a VERY basic client that supports only submitting normal
jobs and doesn't quite have the more modern bits of the protocol like
GRAB_JOB_UNIQ plumbed in yet. A big hole that should be fairly easy to
fill is just making it work with multiple gearman servers (right now it
allows adding many, but only works with the first one added).
However, I've tested them with both gearmand and libgearman, and
everything seems to function at the surface level.
If you want to play, you'll need to build from source, and thus, get a
rusty nightly version up and running (actually it might run on stable
now.. would love to hear if you can get it to).
To the other gearmand maintainers: How would you feel about deprecating
the C++ bits and focusing on this one? Could you make it work for your
org? What features would you need? I *really* don't want to work in
C/C++ anymore. :)
Anyway, thanks for reading, and playing. Cheers!
[1] http://fewbar.com/2017/01/a-love-letter-to-rust/
[2] http://fewbar.com/rust-for-justice/
[3] https://twitter.com/spamaps/status/1245012599873261575
bruce
Apr 16, 2020, 12:42:52 PM4/16/20
to gea...@googlegroups.com
Hi Clint.
Sorry to hear you're laid off :(
Does that mean you're taking a break? Actively looking? Or already on your next gig?
Let us know man! We all need to help each other in this mess!
--
You received this message because you are subscribed to the Google Groups "Gearman" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gearman+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gearman/158705044347.30162.5943445184644122393%40clint-ThinkPad-X1-Carbon-6th.
Edward J. Sabol
Apr 16, 2020, 4:32:37 PM4/16/20
to gea...@googlegroups.com
On Apr 16, 2020, at 11:20 AM, 'Clint Byrum' via Gearman <gea...@googlegroups.com> wrote:
> To the other gearmand maintainers: How would you feel about deprecating
> the C++ bits and focusing on this one? Could you make it work for your
> org? What features would you need?
I'm not really a fan of Rust's syntax ("let" and the exclamation points on macros are huge turn-offs for me). The concept and feature set of Rust is very cool though. I doubt I'd be able to contribute to such a project because I don't know Rust very well. That could change, of course, but that's where I am right now.
> To the other gearmand maintainers: How would you feel about deprecating
> the C++ bits and focusing on this one? Could you make it work for your
> org? What features would you need?
I would be unlikely to switch my organization from using gearmand mainly because gearmand works and fits our requirements fine as-is. I'd probably need a new feature or good reason to switch implementations, like vastly better performance under high loads or something like. And I'd need SSL/TLS support to even consider switching. I can't seem to tell from the rustygeard source code if it currently supports that or not. It doesn't seem to based on my googling of Rust and SSL support, but maybe it's hidden in some Rust macro that I don't understand?
Generally, I would not be in favor of "deprecating" gearmand per se. Public popularity/usage will determine that on it's own. That said, a Rust implementation of Gearman is very cool (and impressively minimalistic in terms of lines of code). Gearman has a long history of being (re-)implemented in multiple languages, and Gearman has always been agnostic where it comes to the implementation language. It would certainly be appropriate to put it in a separate repo under github.com/gearman/, fwiw, if/when you didn't want to just keep it under your personal GitHub area, but I'm probably stating the obvious there.
Regards,
Ed
Алексей Пастухов
Apr 17, 2020, 4:47:27 AM4/17/20
to gea...@googlegroups.com
On Thu, 16 Apr 2020 at 17:20, 'Clint Byrum' via Gearman <gea...@googlegroups.com> wrote:
Greetings Gearman enthusiasts and users!
So, without furter ado, I give you: Rustygear, and Rustygeard
It looks very promising to me.
However, I've tested them with both gearmand and libgearman, and
everything seems to function at the surface level.
I've tested it with perl-Gearman. Looks good.
t/00-use.t ...................... ok
t/01-object.t ................... ok
t/02-client.t ................... ok
t/03-worker.t ................... ok
t/04-task.t ..................... 1/33 Gearman::Task key high_priority is deprecated.
Use priority => "high" instead
Gearman::Task key high_priority is deprecated.
Use priority => "high" instead
t/04-task.t ..................... ok
t/05-taskset.t .................. ok
t/06-response-parser.t .......... ok
t/07-response-parser-taskset.t .. ok
t/08-jobstatus.t ................ ok
t/09-connect.t .................. ok
t/10-job.t ...................... ok
t/11-unit.t ..................... ok
t/12-sum.t ...................... skipped: Can't find gearmand to test with
t/13-fail.t ..................... skipped: Can't find gearmand to test with
t/14-sleep.t .................... skipped: Can't find gearmand to test with
t/15-priority.t ................. skipped: Can't find gearmand to test with
t/16-background.t ............... skipped: Can't find gearmand to test with
t/17-status.t ................... skipped: Can't find gearmand to test with
t/18-ssl.t ...................... skipped: without $ENV{SSL_GEARMAND_HOST}
t/19-add_task-in-cb-ssl.t ....... skipped: without $ENV{SSL_GEARMAND_HOST}
t/19-add_task-in-cb.t ........... skipped: Can't find gearmand to test with
t/20-send-work-command.t ........ skipped: Can't find gearmand to test with
t/25-work-finish-command.t ...... skipped: Can't find gearmand to test with
t/40-prefix.t ................... skipped: Can't find gearmand to test with
t/50-wait_timeout.t ............. skipped: Can't find gearmand to test with
t/65-responseparser.t ........... ok
t/client-on-fail.t .............. skipped: Can't find gearmand to test with
All tests successful.
Files=27, Tests=308, 25 wallclock secs ( 0.11 usr 0.03 sys + 3.82 cusr 0.47 csys = 4.43 CPU)
Result: PASS
To the other gearmand maintainers: How would you feel about deprecating
the C++ bits and focusing on this one? Could you make it work for your
org? What features would you need? I *really* don't want to work in
C/C++ anymore. :)
It's very good to me, because the old code base is absolutely unmaintainable. I'm very sorry for shouting it loudly. But the fact we couldn't build a community around the project even after putting the project on github is clear evidence of that.
Do you consider to move the project into Gearman Job Server Organisation?
Алексей Пастухов
Apr 17, 2020, 7:44:33 AM4/17/20
to gea...@googlegroups.com
On Thu, 16 Apr 2020 at 22:32, Edward J. Sabol <edward...@gmail.com> wrote:
And I'd need SSL/TLS support to even consider switching. I can't seem to tell from the rustygeard source code if it currently supports that or not. It doesn't seem to based on my googling of Rust and SSL support, but maybe it's hidden in some Rust macro that I don't understand?
I guess adding SSL/TLS support to rustygeard is feasible. The crate rusttls seems very promising for the purpose.
Clint Byrum
Apr 18, 2020, 1:33:10 AM4/18/20
to gea...@googlegroups.com
Quoting Edward J. Sabol (2020-04-16 13:32:33)
on the front end. One thing I like about it though, is that generally if
the code compiles, it's got a really good shot of working right.
It's not a short term suggestion btw. But one that I wonder about for
the long term health of the project. I, for one, am kind of over C/C++
now that I've bought in to Rust.
> I would be unlikely to switch my organization from using gearmand mainly because gearmand works and fits our requirements fine as-is. I'd probably need a new feature or good reason to switch implementations, like vastly better performance under high loads or something like. And I'd need SSL/TLS support to even consider switching. I can't seem to tell from the rustygeard source code if it currently supports that or not. It doesn't seem to based on my googling of Rust and SSL support, but maybe it's hidden in some Rust macro that I don't understand?
>
Have not added TLS support yet no. I'd like to though. :)
The main feature is maintainability. Once something *is* in Rust,
refactors and expansions are extremely ergonomic, and generally
performance regressions don't happen unless one really screws stuff up.
In my limited scale tests though, rustygeard keeps up with gearmand just
fine on one thread. I haven't had time to push it beyond that.
> Generally, I would not be in favor of "deprecating" gearmand per se. Public popularity/usage will determine that on it's own. That said, a Rust implementation of Gearman is very cool (and impressively minimalistic in terms of lines of code). Gearman has a long history of being (re-)implemented in multiple languages, and Gearman has always been agnostic where it comes to the implementation language. It would certainly be appropriate to put it in a separate repo under github.com/gearman/, fwiw, if/when you didn't want to just keep it under your personal GitHub area, but I'm probably stating the obvious there.
>
Thanks! It's missing all of the background job durability, but really
Rust is higher level than C, and Gearmand is kinda 90% C, 10% C++. If
Gearmand had been fully converted to C++ I think you'd see less LOC as
well.
> On Apr 16, 2020, at 11:20 AM, 'Clint Byrum' via Gearman <gea...@googlegroups.com> wrote:
> > To the other gearmand maintainers: How would you feel about deprecating
> > the C++ bits and focusing on this one? Could you make it work for your
> > org? What features would you need?
>
> I'm not really a fan of Rust's syntax ("let" and the exclamation points on macros are huge turn-offs for me). The concept and feature set of Rust is very cool though. I doubt I'd be able to contribute to such a project because I don't know Rust very well. That could change, of course, but that's where I am right now.
>
I hear you. Rust is not for everyone, the learning curve is a bit steep
> > To the other gearmand maintainers: How would you feel about deprecating
> > the C++ bits and focusing on this one? Could you make it work for your
> > org? What features would you need?
>
> I'm not really a fan of Rust's syntax ("let" and the exclamation points on macros are huge turn-offs for me). The concept and feature set of Rust is very cool though. I doubt I'd be able to contribute to such a project because I don't know Rust very well. That could change, of course, but that's where I am right now.
>
on the front end. One thing I like about it though, is that generally if
the code compiles, it's got a really good shot of working right.
It's not a short term suggestion btw. But one that I wonder about for
the long term health of the project. I, for one, am kind of over C/C++
now that I've bought in to Rust.
> I would be unlikely to switch my organization from using gearmand mainly because gearmand works and fits our requirements fine as-is. I'd probably need a new feature or good reason to switch implementations, like vastly better performance under high loads or something like. And I'd need SSL/TLS support to even consider switching. I can't seem to tell from the rustygeard source code if it currently supports that or not. It doesn't seem to based on my googling of Rust and SSL support, but maybe it's hidden in some Rust macro that I don't understand?
>
The main feature is maintainability. Once something *is* in Rust,
refactors and expansions are extremely ergonomic, and generally
performance regressions don't happen unless one really screws stuff up.
In my limited scale tests though, rustygeard keeps up with gearmand just
fine on one thread. I haven't had time to push it beyond that.
> Generally, I would not be in favor of "deprecating" gearmand per se. Public popularity/usage will determine that on it's own. That said, a Rust implementation of Gearman is very cool (and impressively minimalistic in terms of lines of code). Gearman has a long history of being (re-)implemented in multiple languages, and Gearman has always been agnostic where it comes to the implementation language. It would certainly be appropriate to put it in a separate repo under github.com/gearman/, fwiw, if/when you didn't want to just keep it under your personal GitHub area, but I'm probably stating the obvious there.
>
Rust is higher level than C, and Gearmand is kinda 90% C, 10% C++. If
Gearmand had been fully converted to C++ I think you'd see less LOC as
well.
Clint Byrum
Apr 18, 2020, 1:34:24 AM4/18/20
to gea...@googlegroups.com
Quoting Алексей Пастухов (2020-04-17 04:44:19)
> crate [2]rusttls seems very promising for the purpose.
>
I'd use this one actually: https://tokio-rs.github.io/tokio-tls/tokio_tls/index.html
> --
> You received this message because you are subscribed to the Google
> Groups "Gearman" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [3]gearman+u...@googlegroups.com.
> ROXy1gwLhc529XLNb63UJVg3Q%40mail.gmail.com.
>
> References
>
> 1. mailto:edward...@gmail.com
> 2. https://crates.io/crates/rustls
> 3. mailto:gearman+u...@googlegroups.com
> 4. https://groups.google.com/d/msgid/gearman/CACULFuDLfOwmEaZwRJuc5hhn...@mail.gmail.com?utm_medium=email&utm_source=footer
>
I'd use this one actually: https://tokio-rs.github.io/tokio-tls/tokio_tls/index.html
> --
> You received this message because you are subscribed to the Google
> Groups "Gearman" group.
> To unsubscribe from this group and stop receiving emails from it, send
> To view this discussion on the web visit
> [4]https://groups.google.com/d/msgid/gearman/CACULFuDLfOwmEaZwRJuc5hhnf
> ROXy1gwLhc529XLNb63UJVg3Q%40mail.gmail.com.
>
> References
>
> 1. mailto:edward...@gmail.com
> 2. https://crates.io/crates/rustls
> 3. mailto:gearman+u...@googlegroups.com
> 4. https://groups.google.com/d/msgid/gearman/CACULFuDLfOwmEaZwRJuc5hhn...@mail.gmail.com?utm_medium=email&utm_source=footer
Clint Byrum
Apr 18, 2020, 1:40:08 AM4/18/20
to gea...@googlegroups.com
Quoting Алексей Пастухов (2020-04-17 01:47:13)
Oh cool, I forgot that this test suite exists. :) Great news that it
passed on the first try. That's as much a testament to the excellent
protocol docs as anything else. :)
>
> To the other gearmand maintainers: How would you feel about
> deprecating
> the C++ bits and focusing on this one? Could you make it work for
> your
> org? What features would you need? I *really* don't want to work in
> C/C++ anymore. :)
>
> It's very good to me, because the old code base is absolutely
> unmaintainable. I'm very sorry for shouting it loudly. But the fact we
> couldn't build a community around the project even after putting the
> project on github is clear evidence of that.
> Do you consider to move the project into [3]Gearman Job Server
> Organisation?
>
I'm just testing the waters right now. gearmand has one huge advantage
over rustygeard at the moment, which is that it is battle tested. But I
agree, I don't really enjoy working in the code, and the C/C++ landscape
has just gotten more confusing lately.
I think if folks are interested in seeing rustygear move into the
gearman org as an experiment, I'd be happy to move it there, especially
if you're up for helping maintain it. Thanks for the patches btw!
For now, let's just let the idea bake in our heads for a bit. Thanks for
your support.
> --
> You received this message because you are subscribed to the Google
> Groups "Gearman" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [4]gearman+u...@googlegroups.com.
> epDUGpuSRg%3DfTti0uf-QM5D2g%40mail.gmail.com.
>
> References
>
> 1. mailto:gea...@googlegroups.com
> 2. https://metacpan.org/release/Gearman
> 3. https://github.com/gearman/
> 4. mailto:gearman+u...@googlegroups.com
> 5. https://groups.google.com/d/msgid/gearman/CACULFuCWEBw5S9OOjLDw6V9ofepDUGpuSRg=fTti0uf...@mail.gmail.com?utm_medium=email&utm_source=footer
> On Thu, 16 Apr 2020 at 17:20, 'Clint Byrum' via Gearman
> <[1]gea...@googlegroups.com> wrote:
>
> Greetings Gearman enthusiasts and users!
> So, without furter ado, I give you: Rustygear, and Rustygeard
>
> It looks very promising to me.
>
> However, I've tested them with both gearmand and libgearman, and
>
> everything seems to function at the surface level.
>
> I've tested it with [2]perl-Gearman. Looks good.
> <[1]gea...@googlegroups.com> wrote:
>
> Greetings Gearman enthusiasts and users!
> So, without furter ado, I give you: Rustygear, and Rustygeard
>
> It looks very promising to me.
>
> However, I've tested them with both gearmand and libgearman, and
>
> everything seems to function at the surface level.
>
passed on the first try. That's as much a testament to the excellent
protocol docs as anything else. :)
>
> To the other gearmand maintainers: How would you feel about
> deprecating
> the C++ bits and focusing on this one? Could you make it work for
> your
> org? What features would you need? I *really* don't want to work in
> C/C++ anymore. :)
>
> It's very good to me, because the old code base is absolutely
> unmaintainable. I'm very sorry for shouting it loudly. But the fact we
> couldn't build a community around the project even after putting the
> project on github is clear evidence of that.
> Organisation?
>
I'm just testing the waters right now. gearmand has one huge advantage
over rustygeard at the moment, which is that it is battle tested. But I
agree, I don't really enjoy working in the code, and the C/C++ landscape
has just gotten more confusing lately.
I think if folks are interested in seeing rustygear move into the
gearman org as an experiment, I'd be happy to move it there, especially
if you're up for helping maintain it. Thanks for the patches btw!
For now, let's just let the idea bake in our heads for a bit. Thanks for
your support.
> --
> You received this message because you are subscribed to the Google
> Groups "Gearman" group.
> To unsubscribe from this group and stop receiving emails from it, send
> To view this discussion on the web visit
> [5]https://groups.google.com/d/msgid/gearman/CACULFuCWEBw5S9OOjLDw6V9of
> epDUGpuSRg%3DfTti0uf-QM5D2g%40mail.gmail.com.
>
> References
>
> 1. mailto:gea...@googlegroups.com
> 2. https://metacpan.org/release/Gearman
> 3. https://github.com/gearman/
> 4. mailto:gearman+u...@googlegroups.com
> 5. https://groups.google.com/d/msgid/gearman/CACULFuCWEBw5S9OOjLDw6V9ofepDUGpuSRg=fTti0uf...@mail.gmail.com?utm_medium=email&utm_source=footer
Clint Byrum
Apr 18, 2020, 1:42:07 AM4/18/20
to gea...@googlegroups.com
Quoting bruce (2020-04-16 09:42:37)
my talents, or know somebody who does, get them into my LinkedIn or
email soon, as I think I'll be moving on to the next thing by the
beginning of May.
https://www.linkedin.com/in/clintbyrum/
> Let us know man! We all need to help each other in this mess!
>
Thanks, it means a lot. I'll be OK, truly, but I definitely can use the
morale boost. :)
> On Thu, Apr 16, 2020 at 11:20 AM 'Clint Byrum' via Gearman
> <[1]gea...@googlegroups.com> wrote:
>
> Greetings Gearman enthusiasts and users!
> Over the past few years, I've been quite enamored with Rust[1][2].
> As a result, I couldn't just sit by idly while gearmand languishes
> in
> C/C++ antiquity.
> So, in my not-quite-ample spare time, I've been writing both a
> server
> and client in Rust.
> With some extra time on my hands since being laid off last month[3],
> I was
> able to get them to a pretty usable spot.
> So, without furter ado, I give you: Rustygear, and Rustygeard
> [2]https://crates.io/crates/rustygear
> [3]https://crates.io/crates/rustygeard
> [2] [5]http://fewbar.com/rust-for-justice/
> [3] [6]https://twitter.com/spamaps/status/1245012599873261575
> 445184644122393%40clint-ThinkPad-X1-Carbon-6th.
> nZXn7dhMw8JhEx4hmxSRdRf5fWaA%40mail.gmail.com.
>
> References
>
> 1. mailto:gea...@googlegroups.com
> 2. https://crates.io/crates/rustygear
> 3. https://crates.io/crates/rustygeard
> 4. http://fewbar.com/2017/01/a-love-letter-to-rust/
> 5. http://fewbar.com/rust-for-justice/
> 6. https://twitter.com/spamaps/status/1245012599873261575
> 7. mailto:gearman+u...@googlegroups.com
> 8. https://groups.google.com/d/msgid/gearman/158705044347.30162.5943445184644122393@clint-ThinkPad-X1-Carbon-6th
> 9. mailto:gearman+u...@googlegroups.com
> 10. https://groups.google.com/d/msgid/gearman/CAP16ngqPFTrjz=Aq3oMCQXnZXn7dhMw8...@mail.gmail.com?utm_medium=email&utm_source=footer
> Hi Clint.
> Sorry to hear you're laid off :(
> Does that mean you're taking a break? Actively looking? Or already on
> your next gig?
I'm actively looking. Got some good leads. If any of you have need for
> Sorry to hear you're laid off :(
> Does that mean you're taking a break? Actively looking? Or already on
> your next gig?
my talents, or know somebody who does, get them into my LinkedIn or
email soon, as I think I'll be moving on to the next thing by the
beginning of May.
https://www.linkedin.com/in/clintbyrum/
> Let us know man! We all need to help each other in this mess!
>
morale boost. :)
> On Thu, Apr 16, 2020 at 11:20 AM 'Clint Byrum' via Gearman
> <[1]gea...@googlegroups.com> wrote:
>
> Greetings Gearman enthusiasts and users!
> Over the past few years, I've been quite enamored with Rust[1][2].
> As a result, I couldn't just sit by idly while gearmand languishes
> in
> C/C++ antiquity.
> So, in my not-quite-ample spare time, I've been writing both a
> server
> and client in Rust.
> With some extra time on my hands since being laid off last month[3],
> I was
> able to get them to a pretty usable spot.
> So, without furter ado, I give you: Rustygear, and Rustygeard
> [3]https://crates.io/crates/rustygeard
> The former is a VERY basic client that supports only submitting
> normal
> jobs and doesn't quite have the more modern bits of the protocol
> like
> GRAB_JOB_UNIQ plumbed in yet. A big hole that should be fairly easy
> to
> fill is just making it work with multiple gearman servers (right now
> it
> allows adding many, but only works with the first one added).
> However, I've tested them with both gearmand and libgearman, and
> everything seems to function at the surface level.
> If you want to play, you'll need to build from source, and thus, get
> a
> rusty nightly version up and running (actually it might run on
> stable
> now.. would love to hear if you can get it to).
> To the other gearmand maintainers: How would you feel about
> deprecating
> the C++ bits and focusing on this one? Could you make it work for
> your
> org? What features would you need? I *really* don't want to work in
> C/C++ anymore. :)
> Anyway, thanks for reading, and playing. Cheers!
> [1] [4]http://fewbar.com/2017/01/a-love-letter-to-rust/
> normal
> jobs and doesn't quite have the more modern bits of the protocol
> like
> GRAB_JOB_UNIQ plumbed in yet. A big hole that should be fairly easy
> to
> fill is just making it work with multiple gearman servers (right now
> it
> allows adding many, but only works with the first one added).
> However, I've tested them with both gearmand and libgearman, and
> everything seems to function at the surface level.
> If you want to play, you'll need to build from source, and thus, get
> a
> rusty nightly version up and running (actually it might run on
> stable
> now.. would love to hear if you can get it to).
> To the other gearmand maintainers: How would you feel about
> deprecating
> the C++ bits and focusing on this one? Could you make it work for
> your
> org? What features would you need? I *really* don't want to work in
> C/C++ anymore. :)
> Anyway, thanks for reading, and playing. Cheers!
> [2] [5]http://fewbar.com/rust-for-justice/
> [3] [6]https://twitter.com/spamaps/status/1245012599873261575
> --
> You received this message because you are subscribed to the Google
> Groups "Gearman" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to [7]gearman+u...@googlegroups.com.
> You received this message because you are subscribed to the Google
> Groups "Gearman" group.
> To unsubscribe from this group and stop receiving emails from it,
> To view this discussion on the web visit
> [8]https://groups.google.com/d/msgid/gearman/158705044347.30162.5943
> 445184644122393%40clint-ThinkPad-X1-Carbon-6th.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Gearman" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [9]gearman+u...@googlegroups.com.
> --
> You received this message because you are subscribed to the Google
> Groups "Gearman" group.
> To unsubscribe from this group and stop receiving emails from it, send
> To view this discussion on the web visit
> [10]https://groups.google.com/d/msgid/gearman/CAP16ngqPFTrjz%3DAq3oMCQX
> nZXn7dhMw8JhEx4hmxSRdRf5fWaA%40mail.gmail.com.
>
> References
>
> 1. mailto:gea...@googlegroups.com
> 2. https://crates.io/crates/rustygear
> 3. https://crates.io/crates/rustygeard
> 4. http://fewbar.com/2017/01/a-love-letter-to-rust/
> 5. http://fewbar.com/rust-for-justice/
> 6. https://twitter.com/spamaps/status/1245012599873261575
> 7. mailto:gearman+u...@googlegroups.com
> 8. https://groups.google.com/d/msgid/gearman/158705044347.30162.5943445184644122393@clint-ThinkPad-X1-Carbon-6th
> 9. mailto:gearman+u...@googlegroups.com
> 10. https://groups.google.com/d/msgid/gearman/CAP16ngqPFTrjz=Aq3oMCQXnZXn7dhMw8...@mail.gmail.com?utm_medium=email&utm_source=footer
dormando
May 1, 2020, 9:57:08 PM5/1/20
to 'Clint Byrum' via Gearman
Hey,
From long term silence, comes an unwanted/uninvited opinion :)
> I hear you. Rust is not for everyone, the learning curve is a bit steep
> on the front end. One thing I like about it though, is that generally if
> the code compiles, it's got a really good shot of working right.
>
> It's not a short term suggestion btw. But one that I wonder about for
> the long term health of the project. I, for one, am kind of over C/C++
> now that I've bought in to Rust.
>
> > I would be unlikely to switch my organization from using gearmand mainly because gearmand works and fits our requirements fine as-is. I'd probably need a new feature or good reason to switch implementations, like vastly better performance under high loads or something like. And I'd need SSL/TLS support to even consider switching. I can't seem to tell from the rustygeard source code if it currently supports that or not. It doesn't seem to based on my googling of Rust and SSL support, but maybe it's hidden in some Rust macro that I don't understand?
> >
>
> Have not added TLS support yet no. I'd like to though. :)
>
> The main feature is maintainability. Once something *is* in Rust,
> refactors and expansions are extremely ergonomic, and generally
> performance regressions don't happen unless one really screws stuff up.
>
> In my limited scale tests though, rustygeard keeps up with gearmand just
> fine on one thread. I haven't had time to push it beyond that.
>
> > Generally, I would not be in favor of "deprecating" gearmand per se. Public popularity/usage will determine that on it's own. That said, a Rust implementation of Gearman is very cool (and impressively minimalistic in terms of lines of code). Gearman has a long history of being (re-)implemented in multiple languages, and Gearman has always been agnostic where it comes to the implementation language. It would certainly be appropriate to put it in a separate repo under github.com/gearman/, fwiw, if/when you didn't want to just keep it under your personal GitHub area, but I'm probably stating the obvious there.
> >
>
> Thanks! It's missing all of the background job durability, but really
> Rust is higher level than C, and Gearmand is kinda 90% C, 10% C++. If
> Gearmand had been fully converted to C++ I think you'd see less LOC as
> well.
This is a neat project (need to say this first!). I want to echo (echo?)
that the "rewrite in Rust" stuff is obnoxious on its own. I struggle a bit
with the language since most of what I do is data structures, which it
seems to do nothing but make difficult. If you can use a library and get
higher level than that it seems alright?
I have a point tho: it'd be nice if this came with an engineering goal.
Being in another language is maintenance; not a real goal to move the
project forward. Else your end goal is only to get back to where you are
now. That sucks.
If I dig my arms into history I'd welcome a chance to focus a
simplification of the daemon. The job durability stuff adds a lot to the
daemon itself with varied guarantees. I really liked the old Garivini
code; sits outside gearman, gets better with small protocol tweaks, etc.
Let the server be narrow and scalable, and let garivini persistence
daemons be written in any language that speaks gearman. It also scales
much better since you have a separation of duties between gearmand and the
persistence systems. You don't need 10 different rust clients for the
various persistence systems.
There isn't really much _to_ gearmand beyond the core protocol and a few
basic optimizations otherwise:
1) accept jobs from clients
2) wake up N workers, then All workers after a timeout (main
optimization!)
3) synchronous jobs
4) scheduled jobs
5) mild job priority system
That's it. Sprinkle TLS on top and you're done. If I wanted to innovate
I'd add batch ops to the protocol (which helps Garivini), unless I missed
that being added to the C daemon at some point :)
-Dormando
From long term silence, comes an unwanted/uninvited opinion :)
> I hear you. Rust is not for everyone, the learning curve is a bit steep
> on the front end. One thing I like about it though, is that generally if
> the code compiles, it's got a really good shot of working right.
>
> It's not a short term suggestion btw. But one that I wonder about for
> the long term health of the project. I, for one, am kind of over C/C++
> now that I've bought in to Rust.
>
> > I would be unlikely to switch my organization from using gearmand mainly because gearmand works and fits our requirements fine as-is. I'd probably need a new feature or good reason to switch implementations, like vastly better performance under high loads or something like. And I'd need SSL/TLS support to even consider switching. I can't seem to tell from the rustygeard source code if it currently supports that or not. It doesn't seem to based on my googling of Rust and SSL support, but maybe it's hidden in some Rust macro that I don't understand?
> >
>
> Have not added TLS support yet no. I'd like to though. :)
>
> The main feature is maintainability. Once something *is* in Rust,
> refactors and expansions are extremely ergonomic, and generally
> performance regressions don't happen unless one really screws stuff up.
>
> In my limited scale tests though, rustygeard keeps up with gearmand just
> fine on one thread. I haven't had time to push it beyond that.
>
> > Generally, I would not be in favor of "deprecating" gearmand per se. Public popularity/usage will determine that on it's own. That said, a Rust implementation of Gearman is very cool (and impressively minimalistic in terms of lines of code). Gearman has a long history of being (re-)implemented in multiple languages, and Gearman has always been agnostic where it comes to the implementation language. It would certainly be appropriate to put it in a separate repo under github.com/gearman/, fwiw, if/when you didn't want to just keep it under your personal GitHub area, but I'm probably stating the obvious there.
> >
>
> Thanks! It's missing all of the background job durability, but really
> Rust is higher level than C, and Gearmand is kinda 90% C, 10% C++. If
> Gearmand had been fully converted to C++ I think you'd see less LOC as
> well.
that the "rewrite in Rust" stuff is obnoxious on its own. I struggle a bit
with the language since most of what I do is data structures, which it
seems to do nothing but make difficult. If you can use a library and get
higher level than that it seems alright?
I have a point tho: it'd be nice if this came with an engineering goal.
Being in another language is maintenance; not a real goal to move the
project forward. Else your end goal is only to get back to where you are
now. That sucks.
If I dig my arms into history I'd welcome a chance to focus a
simplification of the daemon. The job durability stuff adds a lot to the
daemon itself with varied guarantees. I really liked the old Garivini
code; sits outside gearman, gets better with small protocol tweaks, etc.
Let the server be narrow and scalable, and let garivini persistence
daemons be written in any language that speaks gearman. It also scales
much better since you have a separation of duties between gearmand and the
persistence systems. You don't need 10 different rust clients for the
various persistence systems.
There isn't really much _to_ gearmand beyond the core protocol and a few
basic optimizations otherwise:
1) accept jobs from clients
2) wake up N workers, then All workers after a timeout (main
optimization!)
3) synchronous jobs
4) scheduled jobs
5) mild job priority system
That's it. Sprinkle TLS on top and you're done. If I wanted to innovate
I'd add batch ops to the protocol (which helps Garivini), unless I missed
that being added to the C daemon at some point :)
-Dormando
Clint Byrum
May 3, 2020, 2:07:35 PM5/3/20
to Gearman
On Friday, May 1, 2020 at 6:57:08 PM UTC-7, Dormando wrote:
Hey,
From long term silence, comes an unwanted/uninvited opinion :)
Great to see you here. :)
> I hear you. Rust is not for everyone, the learning curve is a bit steep
> on the front end. One thing I like about it though, is that generally if
> the code compiles, it's got a really good shot of working right.
>
> It's not a short term suggestion btw. But one that I wonder about for
> the long term health of the project. I, for one, am kind of over C/C++
> now that I've bought in to Rust.
>
> > I would be unlikely to switch my organization from using gearmand mainly because gearmand works and fits our requirements fine as-is. I'd probably need a new feature or good reason to switch implementations, like vastly better performance under high loads or something like. And I'd need SSL/TLS support to even consider switching. I can't seem to tell from the rustygeard source code if it currently supports that or not. It doesn't seem to based on my googling of Rust and SSL support, but maybe it's hidden in some Rust macro that I don't understand?
> >
>
> Have not added TLS support yet no. I'd like to though. :)
>
> The main feature is maintainability. Once something *is* in Rust,
> refactors and expansions are extremely ergonomic, and generally
> performance regressions don't happen unless one really screws stuff up.
>
> In my limited scale tests though, rustygeard keeps up with gearmand just
> fine on one thread. I haven't had time to push it beyond that.
>
> > Generally, I would not be in favor of "deprecating" gearmand per se. Public popularity/usage will determine that on it's own. That said, a Rust implementation of Gearman is very cool (and impressively minimalistic in terms of lines of code). Gearman has a long history of being (re-)implemented in multiple languages, and Gearman has always been agnostic where it comes to the implementation language. It would certainly be appropriate to put it in a separate repo under github.com/gearman/, fwiw, if/when you didn't want to just keep it under your personal GitHub area, but I'm probably stating the obvious there.
> >
>
> Thanks! It's missing all of the background job durability, but really
> Rust is higher level than C, and Gearmand is kinda 90% C, 10% C++. If
> Gearmand had been fully converted to C++ I think you'd see less LOC as
> well.
This is a neat project (need to say this first!). I want to echo (echo?)
that the "rewrite in Rust" stuff is obnoxious on its own. I struggle a bit
with the language since most of what I do is data structures, which it
seems to do nothing but make difficult. If you can use a library and get
higher level than that it seems alright?
Thanks! Basically, this is the only way I want to play with gearman, so it's a community-interest effort, though I hope one that will open doors to engineering.
Regarding data structures in rust.. I'm sorry that you've had frustration. If you wade back in, check out unsafe, which is specifically *for people like you*. The point of Rust is to default to safety so it feels like a high-level language 90% of the time, but to let you write small sections of code with your own algorithmic assurances and testing. Seriously. That said, without much state to manage, I don't think gearman needs any complex structures that aren't available in libraries already. It's mostly hashmaps and vecs. :-P
I have a point tho: it'd be nice if this came with an engineering goal.
Being in another language is maintenance; not a real goal to move the
project forward. Else your end goal is only to get back to where you are
now. That sucks.
If somebody has an engineering goal they should definitely work on it. What I see is that gearman languishes because nobody with an engineering problem to solve wants to work in hybrid C/C++. So, from an engineering perspective you are right: rustygear wants to match gearmand, not exceed it. But from a community standpoint, rustygear wants to open the door to folks using and contributing more. Also, rust does offer a better security story.
If I dig my arms into history I'd welcome a chance to focus a
simplification of the daemon. The job durability stuff adds a lot to the
daemon itself with varied guarantees. I really liked the old Garivini
code; sits outside gearman, gets better with small protocol tweaks, etc.
Let the server be narrow and scalable, and let garivini persistence
daemons be written in any language that speaks gearman. It also scales
much better since you have a separation of duties between gearmand and the
persistence systems. You don't need 10 different rust clients for the
various persistence systems.
I have definitely thought about turning attention to building a built-in scale-out system like Garivini in Rust. But.. this is just a labor of love, and I'm wary of spending too much time chasing my own desires without any real world tether. :)
There isn't really much _to_ gearmand beyond the core protocol and a few
basic optimizations otherwise:
1) accept jobs from clients
2) wake up N workers, then All workers after a timeout (main
optimization!)
OOPS! I forgot about this one. Thanks, I'll add it to rustygeard.
3) synchronous jobs
4) scheduled jobs
This never worked in the C gearmand. The one reason I don't like it in the gearman daemon is that it represents state, just like background jobs. Much better handled by scaled out workers IMO.
5) mild job priority system
That's it. Sprinkle TLS on top and you're done. If I wanted to innovate
I'd add batch ops to the protocol (which helps Garivini), unless I missed
that being added to the C daemon at some point :)
Yeah that'd be kind of interesting.
I also kind of think making an HTTP/2 version of the protocol that would open up gearman to all languages would be an intersting thing.
Really though, utlimately, it just feels that folks aren't seeing a need for a gearmand anymore. Folks use redis or rabbitmq or one of the cloud-provided things like SQS for job queueing.. and if you don't need the coalescing features of gearmand.. it doesn't offer you much over a much more mature and featureful thing like redis.
Thanks for weighing in!
-Dormando
dormando
May 3, 2020, 2:59:05 PM5/3/20
to 'Clint Byrum' via Gearman
> Regarding data structures in rust.. I'm sorry that you've had frustration. If you wade back in, check out unsafe, which is specifically *for people like
> you*. The point of Rust is to default to safety so it feels like a high-level language 90% of the time, but to let you write small sections of code with
> your own algorithmic assurances and testing. Seriously. That said, without much state to manage, I don't think gearman needs any complex structures that
> aren't available in libraries already. It's mostly hashmaps and vecs. :-P
Given the abuse I've seen thrown at users of unsafe I think the rust
> you*. The point of Rust is to default to safety so it feels like a high-level language 90% of the time, but to let you write small sections of code with
> your own algorithmic assurances and testing. Seriously. That said, without much state to manage, I don't think gearman needs any complex structures that
> aren't available in libraries already. It's mostly hashmaps and vecs. :-P
community can take some time to mature before I wade into it seriously. I
have a laundry list of complaints and too much work to do, but I don't
want that to take away from anyone who actually enjoys using it.
> If somebody has an engineering goal they should definitely work on it. What I see is that gearman languishes because nobody with an engineering problem to
> solve wants to work in hybrid C/C++. So, from an engineering perspective you are right: rustygear wants to match gearmand, not exceed it. But from a
> community standpoint, rustygear wants to open the door to folks using and contributing more. Also, rust does offer a better security story.
and C gearmand in the gearman repo, I'd see a C daemon with a ton of
features (mostly various storage methods) and an MVP in rust. I doubt
anyone/many people would want to dive in as either a user or developer
with that much gap.
By clarifing that you have a goal of not including most of those C
features (and why) you go from a toy MVP to nearly complete.
> I have definitely thought about turning attention to building a built-in scale-out system like Garivini in Rust. But.. this is just a labor of love, and
> I'm wary of spending too much time chasing my own desires without any real world tether. :)
advertise it's a thing and that (when you're ready) you consider
rustygear feature complete; someone could go write the connector in Go or
C or Brainfuck or Rust or use the perl version. You're done already.
IIRC pledging to a storage scaleout model of Garivini only means some
small things for the main daemon:
1) the "status" command needs to be fast (or a fast command to list number
of jobs/bytes in each queue). It wasn't fast in one of the daemons and I
never fixed that.
2) Consider batch jobs, but that's purely an optimization.
3) You're done. You personally don't have to write any other code.
> 2) wake up N workers, then All workers after a timeout (main
> optimization!)
>
>
> OOPS! I forgot about this one. Thanks, I'll add it to rustygeard.
which ones it woke up first (first to check in? last to check in?). This
confused some users since it wouldn't "distribute" load unless you
actually had load.
> This never worked in the C gearmand. The one reason I don't like it in the gearman daemon is that it represents state, just like background jobs. Much
> better handled by scaled out workers IMO.
> Really though, utlimately, it just feels that folks aren't seeing a need for a gearmand anymore. Folks use redis or rabbitmq or one of the cloud-provided
> things like SQS for job queueing.. and if you don't need the coalescing features of gearmand.. it doesn't offer you much over a much more mature and
> featureful thing like redis.
in :) I wish we blogged about gearman harder back in the day. I remember
Starling and Gearman existing at the same time (starling coming after),
and it was such a piece of trash in comparison. Queue software evolved
from starling (rescue/etc) and we did the world a disservice :(
I do say though that it's easy to kick off a queue in redis but it doesn't
have a concept of a distributed system at all. that's what gearman was
from the start and continues to provide. The internal job persistence
plugins took away from that and made it feel more pointless.
but.. yeah, most people just use whatever the cloud has now :/ oh well.
Thanks a ton for keeping things alive!
-Dormando
Clint Byrum
May 3, 2020, 11:25:38 PM5/3/20
to gea...@googlegroups.com
On Sun, May 3, 2020, 11:59 AM dormando <dorm...@rydia.net> wrote:
> Regarding data structures in rust.. I'm sorry that you've had frustration. If you wade back in, check out unsafe, which is specifically *for people like
> you*. The point of Rust is to default to safety so it feels like a high-level language 90% of the time, but to let you write small sections of code with
> your own algorithmic assurances and testing. Seriously. That said, without much state to manage, I don't think gearman needs any complex structures that
> aren't available in libraries already. It's mostly hashmaps and vecs. :-P
Given the abuse I've seen thrown at users of unsafe I think the rust
community can take some time to mature before I wade into it seriously. I
have a laundry list of complaints and too much work to do, but I don't
want that to take away from anyone who actually enjoys using it.
Ugh yeah, it goes both ways, right?
> If somebody has an engineering goal they should definitely work on it. What I see is that gearman languishes because nobody with an engineering problem to
> solve wants to work in hybrid C/C++. So, from an engineering perspective you are right: rustygear wants to match gearmand, not exceed it. But from a
> community standpoint, rustygear wants to open the door to folks using and contributing more. Also, rust does offer a better security story.
My communication skills are waning :) If I'm an outsider and see rustygear
and C gearmand in the gearman repo, I'd see a C daemon with a ton of
features (mostly various storage methods) and an MVP in rust. I doubt
anyone/many people would want to dive in as either a user or developer
with that much gap.
By clarifing that you have a goal of not including most of those C
features (and why) you go from a toy MVP to nearly complete.
Thanks, that is in fact a good idea. We'll see how much time I have to do it with my new position. 😁
> I have definitely thought about turning attention to building a built-in scale-out system like Garivini in Rust. But.. this is just a labor of love, and
> I'm wary of spending too much time chasing my own desires without any real world tether. :)
You don't have to do anything to support Garivini. You just have to
advertise it's a thing and that (when you're ready) you consider
rustygear feature complete; someone could go write the connector in Go or
C or Brainfuck or Rust or use the perl version. You're done already.
I'd want to have some very simple components that all work together and allow devs to approach the whole problem.
IIRC pledging to a storage scaleout model of Garivini only means some
small things for the main daemon:
1) the "status" command needs to be fast (or a fast command to list number
of jobs/bytes in each queue). It wasn't fast in one of the daemons and I
never fixed that.
I was thinking it'd be a good thing to solve with the workers sending UDP messages to an aggregator a la statsd, rather than interrupt the gearmand to read them.
2) Consider batch jobs, but that's purely an optimization.
3) You're done. You personally don't have to write any other code.
> 2) wake up N workers, then All workers after a timeout (main
> optimization!)
>
>
> OOPS! I forgot about this one. Thanks, I'll add it to rustygeard.
Ah cool heh. I can't fully remember anymore but there was an order to
which ones it woke up first (first to check in? last to check in?). This
confused some users since it wouldn't "distribute" load unless you
actually had load.
I think the perl and C differed here. I can see a couple of options that might be better for different workloads.
> This never worked in the C gearmand. The one reason I don't like it in the gearman daemon is that it represents state, just like background jobs. Much
> better handled by scaled out workers IMO.
No arguments here. Garivini-style can do that trivially.
> Really though, utlimately, it just feels that folks aren't seeing a need for a gearmand anymore. Folks use redis or rabbitmq or one of the cloud-provided
> things like SQS for job queueing.. and if you don't need the coalescing features of gearmand.. it doesn't offer you much over a much more mature and
> featureful thing like redis.
Yeah more or less. Sorry if I contributed to this by not putting any work
in :) I wish we blogged about gearman harder back in the day. I remember
Starling and Gearman existing at the same time (starling coming after),
and it was such a piece of trash in comparison. Queue software evolved
from starling (rescue/etc) and we did the world a disservice :(
I think the explosion of memcached, and the numerous less-perfect solutions to its problems just pushed gearmand out of the consciousness.
I do say though that it's easy to kick off a queue in redis but it doesn't
have a concept of a distributed system at all. that's what gearman was
from the start and continues to provide. The internal job persistence
plugins took away from that and made it feel more pointless.
They served a purpose so I'm still hesitant to drop them but I do wish Garivini style was as simple as they are.
but.. yeah, most people just use whatever the cloud has now :/ oh well.
Thanks a ton for keeping things alive!
It's a service so many have done for me. Happy to keep it going. 👍
-Dormando
--
You received this message because you are subscribed to the Google Groups "Gearman" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gearman+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gearman/alpine.DEB.2.21.2005031144030.9649%40dskull.
Reply all
Reply to author
Forward
0 new messages