Persistence and failure scenarios

615 views
Skip to first unread message

Parand Darugar

unread,
Oct 13, 2008, 1:39:28 PM10/13/08
to beanstalk-talk
Thinking through the need for persistence -

Having a log based system would be useful for things beyond dealing
with the power going out - eg. you could do auditing with logs - but
putting that aside and thinking only of failure scenarios:

I'm wondering what is the real-world probability of power going out -
ie. failures that are truly sudden and can't be detected / dealt with
- vs. failures that can be detected.

Point being, if the case where something goes wrong but the process
still has some control is the more common case, perhaps building a
system to dump state before crashing would provide most of the benefit
of persistence without the performance penalty.

This dump-state / read-state capability would probably the same thing
that'd be used for gracefully shutting down and restarting a
beanstalkd.

I haven't looked at the innards of beanstalkd and have no real-world
experience with it so I have no idea what the common failure scenarios
are, but I thought I'd throw it out there to see what people think.

Best,

Parand

Dustin

unread,
Oct 13, 2008, 3:00:25 PM10/13/08
to beanstalk-talk

On Oct 13, 10:39 am, Parand Darugar <daru...@gmail.com> wrote:
> Thinking through the need for persistence -

This may or may not be useful for you:

http://www.rockstarprogrammer.org/post/2008/oct/04/what-matters-asynchronous-job-queue/

Basically, queue durability is somewhat important, but job
durability is far more important. I've lost more jobs after they
leave the queue than I ever have within beanstalkd.

Parand Darugar

unread,
Oct 13, 2008, 3:55:38 PM10/13/08
to beanstalk-talk
Hi Dustin,

Thanks, that's very helpful. It does read like an ad for beanstalkd
though, your requirements line up pretty magically ;-)

Blaine's response is interesting: "it has often been advantageous to
stop a starling server and restart it later". I claim this supports
the argument that dump-state is a useful feature. In the spectrum of
value vs. effort, I agree with your arguments that beanstalkd is at a
sweet spot, and claim that the next milepost could be dump-state (as
opposed to going full on persistence).

Encouraging to hear that "[queue duarbility] isn't what's ever caused
me to lose a job" . I take it beanstalkd is quite stable.

Best,

Parand

On Oct 13, 12:00 pm, Dustin <dsalli...@gmail.com> wrote:
> On Oct 13, 10:39 am, Parand Darugar <daru...@gmail.com> wrote:
>
> > Thinking through the need for persistence -
>
> This may or may not be useful for you:
>
> http://www.rockstarprogrammer.org/post/2008/oct/04/what-matters-async...

Dustin

unread,
Oct 13, 2008, 5:20:34 PM10/13/08
to beanstalk-talk

On Oct 13, 12:55 pm, Parand Darugar <daru...@gmail.com> wrote:

> Thanks, that's very helpful. It does read like an ad for beanstalkd
> though, your requirements line up pretty magically ;-)

Yeah, I am a fan...

> Blaine's response is interesting: "it has often been advantageous to
> stop a starling server and restart it later". I claim this supports
> the argument that dump-state is a useful feature. In the spectrum of
> value vs. effort, I agree with your arguments that beanstalkd is at a
> sweet spot, and claim that the next milepost could be dump-state (as
> opposed to going full on persistence).

You can dump the state through the front-end protocol. It's not
*hugely* efficient, but it works fine. If you put the server in drain
mode, it's trivial to write a tool to migrate all jobs from one server
to another -- you'll lose a little bit of job stats, but the jobs
themselves can be safely picked up and moved.

> Encouraging to hear that "[queue duarbility] isn't what's ever caused
> me to lose a job" . I take it beanstalkd is quite stable.

It is. I've crashed it with my own work in process, but it's only
gone down altogether on dev systems when someone turns off a machine.

Queue durability comes at a cost you have to pay for every single
job, so it does need to be done carefully. I'm not *against* it, but
the main problem I've been having lately is people keep writing queues
against the memcached protocol where the jobs are really easily lost
*and* you have to poll.

Jim Jones

unread,
Oct 14, 2008, 11:35:10 PM10/14/08
to beansta...@googlegroups.com
Hi all,

On Mon, 2008-10-13 at 10:39 -0700, Parand Darugar wrote:
> Thinking through the need for persistence -
>
> Having a log based system would be useful for things beyond dealing
> with the power going out - eg. you could do auditing with logs - but
> putting that aside and thinking only of failure scenarios:
>
> I'm wondering what is the real-world probability of power going out -
> ie. failures that are truly sudden and can't be detected / dealt with
> - vs. failures that can be detected.

Ever heard of Murphy's Law?

> Point being, if the case where something goes wrong but the process
> still has some control is the more common case, perhaps building a
> system to dump state before crashing would provide most of the benefit
> of persistence without the performance penalty.

This may be useful for a controlled shutdown and restart without losing
jobs. But there are hardly any real-world failure conditions that can be
safely trapped. What happens during a hardware failure is pretty much
undefined, at least on the standard hardware that we're all using.

> This dump-state / read-state capability would probably the same thing
> that'd be used for gracefully shutting down and restarting a
> beanstalkd.
>
> I haven't looked at the innards of beanstalkd and have no real-world
> experience with it so I have no idea what the common failure scenarios
> are, but I thought I'd throw it out there to see what people think.

Well, I'd prefer to see the development resources focussed on full blown
persistence, as by the roadmap. Real persistence will also enable
dump/restore but more importantly will it mitigate the reliability
issue.

Alternatively (or additionally) I could imagine some sort of tandem or
"cluster"-mode where all jobs are sent to at least two beanstalkd
instances that live on separate servers. Only one instance (the
"master") would actually be processing jobs, the others would
serve as hot-standby and pick up in case of failure.

The ultimate goal should be to make the queue safe for true "fire and
forget" operation from the job-producer's point of view. Once a job has
entered the queue, it is the queue's responsiblity to keep it safe until
it is consumed - even in the face of hardware failure.


regards, jj


gbarr

unread,
Nov 3, 2008, 3:27:09 PM11/3/08
to beanstalk-talk
On Oct 14, 9:35 pm, Jim Jones <jjimjjo...@googlemail.com> wrote:

> Well, I'd prefer to see the development resources focussed on full blownpersistence, as by the roadmap. Realpersistencewill also enable
> dump/restore but more importantly will it mitigate the reliability
> issue.

Over the weekend I started playing with the code to see what it would
take to add persistence.

My approach was basically a binlog where we just write job structs.
This makes the log files bigger than they need to be, but it is very
cheap to write the file. If/when the server is restarted it would then
read and replay the log files, leaving the in-memory queue in the same
state as when the server died as if all the clients disconnected. ie
there would be no reserved jobs.

> Alternatively (or additionally) I could imagine some sort of tandem or
> "cluster"-mode where all jobs are sent to at least two beanstalkd
> instances that live on separate servers. Only one instance (the
> "master") would actually be processing jobs, the others would
> serve as hot-standby and pick up in case of failure.

I was thinking the same thing. If writing the binlog was actually done
by a slave process then that process could save the binlog and also
copy it to other machines without effecting the performance of the
main server. Those machines could replay the log so that they would
always be ready to take over.

> The ultimate goal should be to make the queue safe for true "fire and
> forget" operation from the job-producer's point of view. Once a job has
> entered the queue, it is the queue's responsiblity to keep it safe until
> it is consumed - even in the face of hardware failure.

Right

Graham.

Graham Barr

unread,
Nov 4, 2008, 4:35:16 PM11/4/08
to beansta...@googlegroups.com
On Nov 3, 2008, at 2:27 PM, gbarr wrote:
> On Oct 14, 9:35 pm, Jim Jones <jjimjjo...@googlemail.com> wrote:
>
>> Well, I'd prefer to see the development resources focussed on full
>> blownpersistence, as by the roadmap. Realpersistencewill also enable
>> dump/restore but more importantly will it mitigate the reliability
>> issue.
>
> Over the weekend I started playing with the code to see what it would
> take to add persistence.
>
> My approach was basically a binlog where we just write job structs.
> This makes the log files bigger than they need to be, but it is very
> cheap to write the file. If/when the server is restarted it would then
> read and replay the log files, leaving the in-memory queue in the same
> state as when the server died as if all the clients disconnected. ie
> there would be no reserved jobs.

For those interested in taking a look I have created a fork on github
with my changes.

http://github.com/gbarr/beanstalkd/commits/binlog

It still requires more work, but it does work. However I have not done
any performance testing.

Graham.

Keith Rarick

unread,
Nov 5, 2008, 3:26:12 AM11/5/08
to beansta...@googlegroups.com
On Tue, Nov 4, 2008 at 1:35 PM, Graham Barr <gmb...@gmail.com> wrote:
> For those interested in taking a look I have created a fork on github
> with my changes.

At first glance I like this a lot. It's almost exactly what I was
thinking of doing, plus the brilliant idea of using ref counts to
expire old files.

The copyright notice in binlog.c lists Keith Rarick and Philotic as
copyright holders, but you wrote the code in there -- it should list
only Graham Barr! And you probably want to change the year to 2008.

I'd love to get this in to 1.2. I'll read it more carefully tomorrow,
do some testing, and possibly push some work onto the binlog branch,
right after I get 1.1 out the door.

kr

Graham Barr

unread,
Nov 5, 2008, 10:11:38 AM11/5/08
to beansta...@googlegroups.com
On Nov 5, 2008, at 2:26 AM, Keith Rarick wrote:
> On Tue, Nov 4, 2008 at 1:35 PM, Graham Barr <gmb...@gmail.com> wrote:
>> For those interested in taking a look I have created a fork on github
>> with my changes.
>
> At first glance I like this a lot. It's almost exactly what I was
> thinking of doing, plus the brilliant idea of using ref counts to
> expire old files.

Good. I think there is a lot we can build on top of this.

> The copyright notice in binlog.c lists Keith Rarick and Philotic as
> copyright holders, but you wrote the code in there -- it should list
> only Graham Barr! And you probably want to change the year to 2008.

OK, I changed that.

> I'd love to get this in to 1.2. I'll read it more carefully tomorrow,
> do some testing, and possibly push some work onto the binlog branch,
> right after I get 1.1 out the door.

It has been a while since I have written any C code, so be gentle :-)
I tried to follow your indentation style, but maybe got it wrong at
times.
Do you use gnuindent ? If so you might want to think about adding
a .indent.pro
to the repository for the style you use.

Graham.

Keith Rarick

unread,
Nov 8, 2008, 7:20:31 PM11/8/08
to beansta...@googlegroups.com
On Wed, Nov 5, 2008 at 7:11 AM, Graham Barr <gmb...@gmail.com> wrote:
> I tried to follow your indentation style, but maybe got it wrong at
> times.
> Do you use gnuindent ? If so you might want to think about adding
> a .indent.pro
> to the repository for the style you use.

I don't use it, but I'm happy to add a .indent.pro file.

kr

Keith Rarick

unread,
Nov 8, 2008, 9:00:19 PM11/8/08
to beansta...@googlegroups.com

On second thought, indent seems unable to put code on the same line as
the if statement, such as

if (!j) return NULL;

and that style occurs throughout the code. This seems to make indent
pretty much unusable.

Instead, I'll just document the conventions that exist and leave it up
to humans to do the right thing. :) I'm not inclined to be super-picky
about this stuff. It's easy enough to clean up later if readability is
an issue.

kr

Keith Rarick

unread,
Nov 8, 2008, 9:56:44 PM11/8/08
to beansta...@googlegroups.com
On Wed, Nov 5, 2008 at 7:11 AM, Graham Barr <gmb...@gmail.com> wrote:
> It has been a while since I have written any C code, so be gentle :-)

It looks mostly okay. There are a couple of bugs to fix and a few
things I'd like to change before merging this into master.

> I tried to follow your indentation style, but maybe got it wrong at
> times.

It follows the existing style pretty well, except for a few minor
things. The only one worth mentioning here is that I'd like to keep
lines under 80 colums wide.

kr

Parand Darugar

unread,
Nov 18, 2008, 3:54:29 PM11/18/08
to beanstalk-talk

A quick check in on progress on this - any general timeframe for when
it'll be officially released / ready for use (or is it already)?

Also, is the log format externally digestible? For example, how hard
would it be to write scripts to analyze type and source of messages?

Btw, much thanks for working on persistence, it's going to make
beanstalkd even more applicable.

Best,

Parand

Keith Rarick

unread,
Nov 19, 2008, 2:39:34 AM11/19/08
to beansta...@googlegroups.com
On Tue, Nov 18, 2008 at 12:54 PM, Parand Darugar <dar...@gmail.com> wrote:
> A quick check in on progress on this - any general timeframe for when
> it'll be officially released / ready for use (or is it already)?

I just merged the binlog branch into master and pushed that out. It is
ready for developers and adventurous users to test. Please try it out
and give us feedback, but don't trust your production data to it yet.

As for a timeframe, I can only say it'll be out when it's ready.

> Also, is the log format externally digestible? For example, how hard
> would it be to write scripts to analyze type and source of messages?

The format is very simple, but subject to change until the next
release. We'll have to decide on a policy about binlog format
compatibility between releases. I would prefer to declare that the
format can change incompatibly at any time, so don't rely on it to
store your jobs across beanstalkd upgrades -- it's mostly for crash
recovery. That said, I doubt it'll change very often, and beanstalkd
will tell you if the format has changed and refuse to run (since
there's a format version number in the binlog). That means you can
safely use a binlog across any version of beanstalkd as long as it
lets you.

You can certainly write external tools to analyze the binlog -- the
format is just a version number followed by a sequence of records. But
it'll likely never be documented extensively (unlike the network
protocol), so authors of such tools will just have to look at the code
in binlog.c that reads the file (it's at the top of binlog_replay()
and not that complicated).

> Btw, much thanks for working on persistence, it's going to make
> beanstalkd even more applicable.

Huge thanks to Graham Barr for kick-starting this feature with an
essentially complete implementation. All that's left is a little
polish and a lot of testing.

kr

Parand Darugar

unread,
Nov 21, 2008, 4:36:12 AM11/21/08
to beanstalk-talk

> > Also, is the log format externally digestible? For example, how hard
> > would it be to write scripts to analyze type and source of messages?
>
> You can certainly write external tools to analyze the binlog -- the
> format is just a version number followed by a sequence of records. But
> it'll likely never be documented extensively (unlike the network
> protocol), so authors of such tools will just have to look at the code
> in binlog.c that reads the file (it's at the top of binlog_replay()
> and not that complicated).

In that case, how about a fairly stable api that could be used to
reliably parse the binlog (I suspect there already is one)? That way
analysis tools can be written without having to worry about the
format.

Best,

Parand

Keith Rarick

unread,
Nov 21, 2008, 4:34:31 PM11/21/08
to beansta...@googlegroups.com
On Fri, Nov 21, 2008 at 1:36 AM, Parand Darugar <dar...@gmail.com> wrote:
> In that case, how about a fairly stable api that could be used to
> reliably parse the binlog (I suspect there already is one)? That way
> analysis tools can be written without having to worry about the
> format.

Hmm, there's no API per se. The code to read the binlog format is
really about ten lines. But maybe documentation would be a good idea.
It won't take long and it'll be nice to have for historical binlog
files after the format has changed.

So, let's say you should look in doc/binlog.txt to find the current
format in each release. And look in doc/historical-binlog.txt for old
formats. I'll make sure to update those files if necessary before each
release.

kr

Ilya Grigorik

unread,
Dec 12, 2008, 1:53:22 PM12/12/08
to beanstalk-talk
Hey guys, what are the pros/cons to increasing the size of the binlog
files? By default they're split into 10mb chunk. Is there a reason why
you would make that number lower or higher? (Performance, recovery
time, etc.?)

ig

Graham Barr

unread,
Dec 12, 2008, 3:28:23 PM12/12/08
to beansta...@googlegroups.com

The value 10mb was an arbitrary choice. The original intent was actually
to add a command line option to set the size and just have 10mb as the
default.

Graham.

Keith Rarick

unread,
Dec 12, 2008, 3:47:51 PM12/12/08
to beansta...@googlegroups.com

Having smaller files saves a little disk space on average, since they
can be deleted with finer granularity. But it also ought to make
things a little slower on average, since there are more disk seeks
when creating a new file and during recovery.

I predict these differences will be small for most people, unless you
are pushing the limits of your hardware.

kr

Reply all
Reply to author
Forward
0 new messages