[Samba] frequent tdb corruption

[Adam Thorn]

Hi,

I'm seeing regular tdb corruption; typical log messages are:

tdb(/var/db/samba/sessionid.tdb): tdb_rec_read bad magic 0x42424242 at
offset=672032

tdb(/var/db/samba/connections.tdb): tdb_rec_read bad magic 0x0 at
offset=1111638594

tdb(/var/db/samba/locking.tdb): tdb_rec_read bad magic 0x42424242 at
offset=1034396

which then prevents fileserving from working properly (N.B. the bad
magic is not limited to those three tdbs). At the moment I'm running
Samba 3.6.6 on FreeBSD 9.0, but I've seen exactly the same behaviour
with 3.6.9 and 3.6.13, and also the same behaviour on FreeBSD 9.1 as
well. I also currently have the tdb-1.2.9,1 FreeBSD port installed at
present, but have seen the same problem with tdb-1.2.11,1

I found a few forum posts that suggested setting "use mmap=no" - I have
tried that, but saw no change in behaviour.

Restarting samba invariably clears the problem for a while: sometimes
it's just a few hours before we get further bad magic messages,
sometimes it's continued working fine for ~10 days or so, and pretty
much everything in between. There is no obvious pattern of which tdbs
are corrupting; I've seen pretty much all of them become corrupt over
the last couple of months.

The server has multiple IP addresses which samba listens on; first of
all we just start smbd with

[global]
include = /data/config/samba/servers/%i

and I've attached the result from running testparm on one of those
included files. It's very very slightly redacted to hide IP addresses
and group names. We have another similarly-configured server (FreeBSD
9.0, Samba 3.6.6) with the same pattern of "include a config file
dependent on the IP address the client connects to", and that has been
running smoothly with no problems at all for over a year.

I don't think (but have not absolutely conclusively ruled out) that it's
a hardware problem on the server itself; the samba service (and the
associated IP addresses) is managed by heartbeat, so I've tried running
samba on the two nominally identical servers in the HA cluster - I see
the same problematic behaviour on both nodes.

I've also attached the output of "smbd -b", in case that is informative.

I'm kind of running out of ideas of what to try next; any and all advice
will thus be gratefully received! It's been especially hard to diagnose
because the corruption happens seemingly at random, and I've not been
able to identify a definite action that leads to the errors. (Also,
because it's a production server, I'm not keen to try to deliberately
provoke errors..)

Adam

[Adam Thorn]

> [global]
> include = /data/config/samba/servers/%i
>
> and I've attached the result from running testparm on one of those
> included files.

..that attachment seemed be mislaid, so it's attached here instead!

Adam

[Gerry Reno]

> [global]
> include = /data/config/samba/servers/%i
>
> and I've attached the result from running testparm on one of those

> included files. It's very very slightly redacted to hide IP addresses
> and group names. We have another similarly-configured server (FreeBSD
> 9.0, Samba 3.6.6) with the same pattern of "include a config file
> dependent on the IP address the client connects to", and that has been
> running smoothly with no problems at all for over a year.
>
> I don't think (but have not absolutely conclusively ruled out) that it's
> a hardware problem on the server itself; the samba service (and the
> associated IP addresses) is managed by heartbeat, so I've tried running
> samba on the two nominally identical servers in the HA cluster - I see
> the same problematic behaviour on both nodes.
>
> I've also attached the output of "smbd -b", in case that is informative.
>
> I'm kind of running out of ideas of what to try next; any and all advice
> will thus be gratefully received! It's been especially hard to diagnose
> because the corruption happens seemingly at random, and I've not been
> able to identify a definite action that leads to the errors. (Also,
> because it's a production server, I'm not keen to try to deliberately
> provoke errors..)
>
> Adam
>
>

What type of filesystem are you using? Do you have barriers enabled?

I know in Linux that you should set barrier=1 on the ext3/ext4 filesystem in order to prevent corruption of sam.ldb in
cases of power loss.



--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba

[Timur I. Bakeyev]

I suspect, that ML software removes attachments, so please put
it somewhere on the Web.

Regards,
Timur Bakeyev.

On Tue, May 14, 2013 at 1:57 PM, Adam Thorn <al...@cam.ac.uk> wrote:

>
> > [global]
> > include = /data/config/samba/servers/%i
> >
> > and I've attached the result from running testparm on one of those
> > included files.
>

> ..that attachment seemed be mislaid, so it's attached here instead!
>
> Adam
>

[Adam Thorn]

> What type of filesystem are you using? Do you have barriers enabled?
>
> I know in Linux that you should set barrier=1 on the ext3/ext4 filesystem in order to prevent corruption of sam.ldb in
> cases of power loss.

It's all on ZFS, so I don't believe there's an equivalent setting.

[Adam Thorn]

OK: my samba config is at

http://www-co.ch.cam.ac.uk/geekery/smb.conf

Thanks,

Adam

On Tue, 2013-05-14 at 16:01 +0200, Timur I. Bakeyev wrote:
> I suspect, that ML software removes attachments, so please put
> it somewhere on the Web.
>
>
> Regards,
> Timur Bakeyev.
>
>
> On Tue, May 14, 2013 at 1:57 PM, Adam Thorn <al...@cam.ac.uk> wrote:
>
> > [global]
> > include = /data/config/samba/servers/%i
> >
> > and I've attached the result from running testparm on one of
> those
> > included files.
>
>
> ..that attachment seemed be mislaid, so it's attached here
> instead!
>

[Gerry Reno]

On 05/14/2013 11:17 AM, Adam Thorn wrote:
>> What type of filesystem are you using? Do you have barriers enabled?
>>
>> I know in Linux that you should set barrier=1 on the ext3/ext4 filesystem in order to prevent corruption of sam.ldb in
>> cases of power loss.
> It's all on ZFS, so I don't believe there's an equivalent setting.
>
> Adam
>

https://groups.google.com/a/zfsonlinux.org/forum/#!msg/zfs-discuss/DKweF6DtaMo/zn6WDNV3OIgJ
<https://groups.google.com/a/zfsonlinux.org/forum/#%21msg/zfs-discuss/DKweF6DtaMo/zn6WDNV3OIgJ>

[Andrew Bartlett]

On Tue, 2013-05-14 at 10:59 +0100, Adam Thorn wrote:
> Hi,
>
> I'm seeing regular tdb corruption; typical log messages are:
>
> tdb(/var/db/samba/sessionid.tdb): tdb_rec_read bad magic 0x42424242 at
> offset=672032
>
> tdb(/var/db/samba/connections.tdb): tdb_rec_read bad magic 0x0 at
> offset=1111638594
>
> tdb(/var/db/samba/locking.tdb): tdb_rec_read bad magic 0x42424242 at
> offset=1034396
>
> which then prevents fileserving from working properly (N.B. the bad
> magic is not limited to those three tdbs). At the moment I'm running
> Samba 3.6.6 on FreeBSD 9.0, but I've seen exactly the same behaviour
> with 3.6.9 and 3.6.13, and also the same behaviour on FreeBSD 9.1 as
> well. I also currently have the tdb-1.2.9,1 FreeBSD port installed at
> present, but have seen the same problem with tdb-1.2.11,1

> I don't think (but have not absolutely conclusively ruled out) that it's
> a hardware problem on the server itself; the samba service (and the
> associated IP addresses) is managed by heartbeat, so I've tried running
> samba on the two nominally identical servers in the HA cluster - I see
> the same problematic behaviour on both nodes.

Can you please clarify:

- Is the filesystem on this disk in any way shared?
- Is the block device involved in any way shared?
- Has the server ever had a unexpected poweroff?
- Do Samba processes ever crash?

If the answer is no to all these, then I would strongly suspect a
hardware or OS/kernel issue. TDB is in general a very reliable
database, and these issues are certainly unexpected.

Similarly, if the answer is no unexpected poweroff, then the barrier=1
thing is a red herring, that is about data safety on unexpected
poweroff.

Could you put your TDB files on a different file system, to rule in our
out ZFS (or the glue between FreeBSD and ZFS)?

Thanks,

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org

[Adam Thorn]

Hi Andrew,

> Can you please clarify:
>
> - Is the filesystem on this disk in any way shared?
> - Is the block device involved in any way shared?

I'm not 100% sure what you mean by "shared" in this context, but the
filesystem where the tdbs are stored is: pair of local disks => mirrored
together via zfs => zfs filesystem.

In case it's relevant: the samba config files are stored on a different
filesystem. There are a set of multiple disks which are mirrored over
the network with FreeBSD's HAST (pretty much equivalent to DRBD), and
the HAST devices are then combined into a RAIDZ2 pool. I have recently
had some disk problems with the filesystem where the samba config files
are stored (but no problems with the filesystem which stores the tdbs) -
is there any mechanism whereby e.g. a timeout in smbd trying to read
it's config file could cause problems with the tdbs?

> - Has the server ever had a unexpected poweroff?

No; I installed a different version of FreeBSD (downgraded from 9.1 to
9.0) a week ago to see if it made any difference. The OS partitions were
totally wiped and reinstalled, and since then the server has been
powered-up and stable, and still exhibiting corrupt tdbs.

> - Do Samba processes ever crash?

No.

> If the answer is no to all these, then I would strongly suspect a
> hardware or OS/kernel issue.

I'd have said the same, but I've seen this problem on two (nominally)
identical pieces of hardware (whilst that doesn't rule out hardware, I
think it reduces the likelihood). I do plan to run memtest etc when I
can, though. Also, I'm running the same OS/kernel on another server
which is *almost* identical hardware, and that's been completely
trouble-free for over a year.

> Could you put your TDB files on a different file system, to rule in our
> out ZFS (or the glue between FreeBSD and ZFS)?

I can certainly give that a go!

Thanks for the help,

Adam

[Michael Wood]

Hi

On 16 May 2013 12:15, Adam Thorn <al...@cam.ac.uk> wrote:
> Hi Andrew,
>
>> Can you please clarify:
>>
>> - Is the filesystem on this disk in any way shared?
>> - Is the block device involved in any way shared?
>
> I'm not 100% sure what you mean by "shared" in this context, but the
> filesystem where the tdbs are stored is: pair of local disks => mirrored
> together via zfs => zfs filesystem.
>
> In case it's relevant: the samba config files are stored on a different
> filesystem. There are a set of multiple disks which are mirrored over
> the network with FreeBSD's HAST (pretty much equivalent to DRBD), and
> the HAST devices are then combined into a RAIDZ2 pool.

I think this is what Andrew was asking about.

DRDB, unless you have a cluster filesystem, needs to have only a
single machine accessing the filesystem at a time. HAST's homepage
says the same:

"HAST works in Primary-Secondary (Master-Backup, Master-Slave)
configuration, which means that only one of the cluster nodes can be
active at any given time."

I think Andrew was asking if perhaps you had two machine accessing the
filesystem at once.

Do you still get the corruption with a local ZFS filesystem?

--
Michael Wood <esio...@gmail.com>

[Volker Lendecke]

On Thu, May 16, 2013 at 11:15:51AM +0100, Adam Thorn wrote:
> Hi Andrew,
>
> > Can you please clarify:
> >
> > - Is the filesystem on this disk in any way shared?
> > - Is the block device involved in any way shared?
>
> I'm not 100% sure what you mean by "shared" in this context, but the
> filesystem where the tdbs are stored is: pair of local disks => mirrored
> together via zfs => zfs filesystem.
>
> In case it's relevant: the samba config files are stored on a different
> filesystem. There are a set of multiple disks which are mirrored over
> the network with FreeBSD's HAST (pretty much equivalent to DRBD), and

> the HAST devices are then combined into a RAIDZ2 pool. I have recently
> had some disk problems with the filesystem where the samba config files
> are stored (but no problems with the filesystem which stores the tdbs) -
> is there any mechanism whereby e.g. a timeout in smbd trying to read
> it's config file could cause problems with the tdbs?
>
> > - Has the server ever had a unexpected poweroff?
>
> No; I installed a different version of FreeBSD (downgraded from 9.1 to
> 9.0) a week ago to see if it made any difference. The OS partitions were
> totally wiped and reinstalled, and since then the server has been
> powered-up and stable, and still exhibiting corrupt tdbs.
>
> > - Do Samba processes ever crash?
>
> No.
>
> > If the answer is no to all these, then I would strongly suspect a
> > hardware or OS/kernel issue.
>
> I'd have said the same, but I've seen this problem on two (nominally)
> identical pieces of hardware (whilst that doesn't rule out hardware, I
> think it reduces the likelihood). I do plan to run memtest etc when I
> can, though. Also, I'm running the same OS/kernel on another server
> which is *almost* identical hardware, and that's been completely
> trouble-free for over a year.
>
> > Could you put your TDB files on a different file system, to rule in our
> > out ZFS (or the glue between FreeBSD and ZFS)?
>
> I can certainly give that a go!

"use mmap = no" might provide another data point.

Volker

--
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de, mailto:kon...@sernet.de

[Adam Thorn]

> > In case it's relevant: the samba config files are stored on a
different
> > filesystem. There are a set of multiple disks which are mirrored
over
> > the network with FreeBSD's HAST (pretty much equivalent to DRBD),
and
> > the HAST devices are then combined into a RAIDZ2 pool.
>

> I think this is what Andrew was asking about.
>
> DRDB, unless you have a cluster filesystem, needs to have only a
> single machine accessing the filesystem at a time. HAST's homepage
> says the same:
>
> "HAST works in Primary-Secondary (Master-Backup, Master-Slave)
> configuration, which means that only one of the cluster nodes can be
> active at any given time."
>
> I think Andrew was asking if perhaps you had two machine accessing the
> filesystem at once.

Ah, I see. In which case, the answer is no: it'll never have been
running in a primary/primary configuration. Also, just to reiterate -
although my samba config files are on the HAST, the tdbs are very
definitely local.

> Do you still get the corruption with a local ZFS filesystem?

I can certainly move my samba config to local disk to see if it improves
matters. I'm just curious if there's an obvious problem with having the
config files on the HAST? (given that we take every possible precaution
to avoid primary/primary situations, and our automated monitoring pays
close attention to that).

Thanks,

Adam

[Adam Thorn]

On Thu, 2013-05-16 at 13:29 +0200, Volker Lendecke wrote:

> "use mmap = no" might provide another data point.

Hi Volker,

Thanks, but I tried setting that some time ago - I continued to see tdb
corruption, so have since returned to not explicitly setting any value
for "use mmap".