Re: binlog error message. 'You may be missing data'

[Ian Docherty]

Note, we are using version 1.4.6 of beanstalk, our system admin's tell me that this is the latest version which is packaged. (I don't know why they can't package it themselves)

- icydee

On Monday, 1 October 2012 15:55:36 UTC+1, Ian Docherty wrote:

Recently when starting beanstalkd we get the following error message numerous times.

/usr/bin/beanstalkd: WARNING, EOF while reading version record at /var/lib/beanstalkd/binlog/binlog.203:0. Continuing. You may be missing data.:

Investigating where this error comes from, it seems to be when beanstalkd checks for a version number in the log file, and can't find it.

Is it possible that the binlog has become corrupted?

I mention this because we had a problem recently where some NFS drive errors may have caused it to crash.

What is the best way of recovering from this error? Should the binlog be deleted manually? If we instruct beatstalk to delete all buried jobs, does this also re-create the binlog file?

- icydee

[Ian Docherty]

Sorry, one further fact, we have about 160 binlogs, some are 10Mb, I understand this is a bug that was in version 1.4 (which we are using), perhaps if we clean these up correctly then the problem may go away?

[Keith Rarick]

On Mon, Oct 1, 2012 at 7:55 AM, Ian Docherty <i...@docherty.me> wrote:
> /usr/bin/beanstalkd: WARNING, EOF while reading version record at
> /var/lib/beanstalkd/binlog/binlog.203:0. Continuing. You may be missing
> data.:

This message means the file binlogl.203 is empty.
What's the file size reported by the OS? I'd expect
it to be 0. This means the contents of that file are
already gone, and you might as well delete the file.

> I mention this because we had a problem recently where some NFS
> drive errors may have caused it to crash.

File locking is unreliable over NFS, and beanstalkd
relies on the F_SETLK fcntl flag to ensure that only
one beanstalkd runs at a time.

[Ian Docherty]

On Wednesday, 3 October 2012 20:35:55 UTC+1, Keith Rarick wrote:

On Mon, Oct 1, 2012 at 7:55 AM, Ian Docherty <i...@docherty.me> wrote:
> /usr/bin/beanstalkd: WARNING, EOF while reading version record at
> /var/lib/beanstalkd/binlog/binlog.203:0. Continuing. You may be missing
> data.:

This message means the file binlogl.203 is empty.
What's the file size reported by the OS? I'd expect
it to be 0. This means the contents of that file are
already gone, and you might as well delete the file.

Yes that was the problem. Thanks.

 

> I mention this because we had a problem recently where some NFS
> drive errors may have caused it to crash.

File locking is unreliable over NFS, and beanstalkd
relies on the F_SETLK fcntl flag to ensure that only
one beanstalkd runs at a time.

only the /data directory runs on NFS (which is where our data files are held)

the OS runs on local disk so I assume file locking used by beanstald would

be on there and so should not cause an issue.

 

 

[Ian Docherty]

On Monday, 22 October 2012 11:25:50 UTC+1, Ian Docherty wrote:

On Wednesday, 3 October 2012 20:35:55 UTC+1, Keith Rarick wrote:

On Mon, Oct 1, 2012 at 7:55 AM, Ian Docherty <i...@docherty.me> wrote:
> /usr/bin/beanstalkd: WARNING, EOF while reading version record at
> /var/lib/beanstalkd/binlog/binlog.203:0. Continuing. You may be missing
> data.:

This message means the file binlogl.203 is empty.
What's the file size reported by the OS? I'd expect
it to be 0. This means the contents of that file are
already gone, and you might as well delete the file.

Yes that was the problem. Thanks.

However, we now get the string of errors:-

 /usr/bin/beanstalkd: binlog.c:768 in binlog_init: /var/lib/beanstalkd/binlog/binlog.203: No such file or directory

Beanstalkd appears to be running ok, so it is only a startup issue.

[Keith Rarick]

On Mon, Oct 22, 2012 at 3:25 AM, Ian Docherty <i...@docherty.me> wrote:
> only the /data directory runs on NFS (which is where our data files are
> held)
> the OS runs on local disk so I assume file locking used by beanstald would
> be on there and so should not cause an issue.

The lock file is inside the directory you give to the -b flag.