State of persistence in Beanstalkd
Parand Darugar
I've been out of the loop on beanstalkd development for a while,
wanted to see what the latest is on persistence. Is persistence in the
latest version of beanstalkd from github production ready and
recommended? Apologies if this information exists somewhere already
but I wasn't able to find it. I'd also love a pointer to some usage
guides on how to enable/recover/examine persistence or some test cases
so I can build my own docs.
Much thanks,
Parand
Keith Rarick
> I've been out of the loop on beanstalkd development for a while,
> wanted to see what the latest is on persistence.
I have been busy with a contract project recently, but I am almost
done with it. Then I'll finish up persistence and make a release.
If anyone else wants to work on persistence right away, I am aware of
two things that need to be done:
1. Allow beanstalkd to reserve arbitrary amounts of binlog space.
http://github.com/kr/beanstalkd/issues#issue/14
2. Go over the actual format and make sure it is robust against errors
in the data on disk, especially file truncation.
http://github.com/kr/beanstalkd/issues/#issue/15
3. Add appropriate fsync calls, with optional throttling.
http://github.com/kr/beanstalkd/issues/#issue/16
> Is persistence in the
> latest version of beanstalkd from github production ready and
> recommended?
Not quite, because of the three things above. But if you trust your
disks and power supply and don't generate big queues and don't mind
binlog files being incompatible with the official release, go for it!
:)
> Apologies if this information exists somewhere already
> but I wasn't able to find it. I'd also love a pointer to some usage
> guides on how to enable/recover/examine persistence or some test cases
> so I can build my own docs.
There isn't much documentation. There is one command line option, -b,
that controls binlog use. It takes the name of a directory where you
want beanstalkd to look for and save binlog files.
kr
Nicolas
make persistence ready for production ? :)
On 6 août, 03:24, Keith Rarick <k...@xph.us> wrote:
> > I've been out of the loop on beanstalkd development for a while,
> > wanted to see what the latest is on persistence.
>
> I have been busy with a contract project recently, but I am almost
> done with it. Then I'll finish up persistence and make a release.
>
> If anyone else wants to work on persistence right away, I am aware of
> two things that need to be done:
>
> 2. Go over the actual format and make sure it is robust against errors
>
> 3. Add appropriate fsync calls, with optional throttling.http://github.com/kr/beanstalkd/issues/#issue/16
Graham Barr
>
> How much do I have to donate on http://pledgie.com/campaigns/956 to
> make persistence ready for production ? :)
I still have one outstanding change that needs to be applied
http://github.com/gbarr/beanstalkd/commit/21b67edc2918ad7fadd7d1895df7aa1c4c0101b8
without it binlog replay can fail on a server restart
With that patch applied we have been using the existing binlog quite
successfully
Graham.
Keith Rarick
First, let me apologize for this taking so long. After the commits I
just pushed today, I am happy with the state of the persistence code
and beanstalkd in general and am just about ready to release 1.4.
It'd be really awesome if those using persistence could check out the
head of master from github.com/kr/beanstalkd and give it a spin. And
anyone else who wants to try out a "release candidate", please go
ahead.
I'll wait a couple of days in case anything crops up, and then go
ahead and tag the release, put it up on the web site, and post an
announcement.
Everyone who has contributed to this release and in the past, I can't
thank you enough. This bears repeating in the release announcement for
1.4, but I just wanted to express my gratitude now. This release is
going to be great. It contains a lot of fixes, a huge new feature, and
is our most portable beanstalkd so far. (And it's entirely my fault
that it has taken so long. Again sorry.) Thanks for sticking with me!
Up next... work on 1.5 with named jobs (for deduplication etc),
mailboxes (erlang-style, for job completion status etc), and maybe
other fun features folks come up with. Oh, and I'll finally get to
work on the web site once 1.4 is behind us. Hooray!
kr
igrigorik
Really looking forward to named jobs. :-)
ig
Graham Barr
> First, let me apologize for this taking so long. After the commits I
> just pushed today, I am happy with the state of the persistence code
> and beanstalkd in general and am just about ready to release 1.4.
There is probably a better way then hard coding in the file, but please update
the version number in beanstalkd.spec
Graham.
Keith Rarick
> There is probably a better way then hard coding in the file, but please update
> the version number in beanstalkd.spec
Thanks for reminding me! It's now an autoconf template and included in
the distributed tarball.
I don't know the usual flow for generating rpm files. Hopefully this
will work. I only know of one problem with this approach: the version
string between releases has hyphens in it (like 1.3-55-g9f71603
instead of just 1.3) and rpm forbids this (according to the
documentation). I welcome any suggestions on how to deal with this.
kr
Jeremy Hinegardner
We've been using 'git describe' to populate the release number in our
projects stored in git. That '55' above, is the number of commits since the
v1.3 tag.
jeremy@stiletto:~/repos/git/beanstalkd master % git describe --match v1.1
v1.1-171-g9f71603
jeremy@stiletto:~/repos/git/beanstalkd master % git describe --match v1.2
v1.2-60-g9f71603
jeremy@stiletto:~/repos/git/beanstalkd master % git describe --match v1.3
v1.3-55-g9f71603
So from beanstalkd's perspective, you can use the 1.3 for the @VERSION@ and
then make:
Release: @RELEASE@%{?dist}
and populate @RELEASE@ based upon the output of 'git describe'.
Its not perfect, if you have the same number of commits from a tag in two
different branches, you could have 2 spec files that had the same
name-version-release but would be two different rpms.
Or make the @VERSION@ be the 1.3-55 only, and leave @RELEASE@ to be the
rpm release.
If I get a chance today I may try a few things and send a pull request.
enjoy,
-jeremy
--
========================================================================
Jeremy Hinegardner jer...@hinegardner.org
Jeremy Dunck
<jer...@hinegardner.org> wrote:
...
> jeremy@stiletto:~/repos/git/beanstalkd master % git describe --match v1.1
> v1.1-171-g9f71603
> jeremy@stiletto:~/repos/git/beanstalkd master % git describe --match v1.2
> v1.2-60-g9f71603
> jeremy@stiletto:~/repos/git/beanstalkd master % git describe --match v1.3
> v1.3-55-g9f71603
>
> Its not perfect, if you have the same number of commits from a tag in two
> different branches, you could have 2 spec files that had the same
> name-version-release but would be two different rpms.
Isn't that hash prefix at the end (g9f71603) the current commit? It's
the same in your example because you were trying to demonstrate
git-describe, but in real life, the output of the same git-describe on
two different commits, even with the same # of commits to the same
ancestor tag, would have different hashes.
Right?
Keith Rarick
<jer...@hinegardner.org> wrote:
> So from beanstalkd's perspective, you can use the 1.3 for the @VERSION@ and
> then make:
>
> Release: @RELEASE@%{?dist}
>
> and populate @RELEASE@ based upon the output of 'git describe'.
That is pretty clever, but I suspect that it will cause confusion.
I'm leaning toward replacing the hyphens with pluses, to get 1.3+55+g9f71603.
kr
Dustin
On Oct 2, 1:32 pm, Keith Rarick <k...@xph.us> wrote:
> I'm leaning toward replacing the hyphens with pluses, to get 1.3+55+g9f71603.
because I have to rewrite them all to hyphens again. That's an
amazingly easy fix in git, though. I should really do that...