Write barriers on MD RAID1

[Ken Milmore]

Support for write barriers on MD RAID1 devices appears to be broken on
2.6.29 and later kernels. Mounting an ext3 or ext4 filesystem over an
md raid1 block device gives the kernel message:

JBD: barrier-based sync failed on md0 - disabling barriers

Is this intentional? It looks like a regression to me, since write
barriers (ostensibly) worked for MD RAID1 on earlier kernels (well, all
of the 2.6.27 series at least...)

Doing a git bisect suggested that this is the culprit:

commit cec0707e40ae25794b5a2de7b7f03c51961f80d9
Author: Jens Axboe <jens....@oracle.com>
Date: Tue Jan 13 15:28:32 2009 +0100

block: silently error an unsupported barrier bio

Rolling this back from 2.6.29.4 made the JBD error message go away for
me. I tried to do some disk benchmarking to determine if write barriers
were really still working, but the results were not very conclusive.

Details of test setup:
- 2.6.29.4 kernel (config from slackware-current)
- ext3 filesystem (mounted with barrier=1), also tried ext4
- MD RAID1 block device over two SATA disks
- Intel motherboard AHCI controller (82801)

I got similar behaviour on a virtualised setup using qemu, with the RAID
over two emulated IDE disks.

Please contact me if I can provide any further information.

-Ken Milmore.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

[NeilBrown]

On Sun, May 24, 2009 9:54 am, Ken Milmore wrote:
>
> Support for write barriers on MD RAID1 devices appears to be broken on
> 2.6.29 and later kernels. Mounting an ext3 or ext4 filesystem over an
> md raid1 block device gives the kernel message:
>
> JBD: barrier-based sync failed on md0 - disabling barriers
>
> Is this intentional? It looks like a regression to me, since write
> barriers (ostensibly) worked for MD RAID1 on earlier kernels (well, all
> of the 2.6.27 series at least...)
>
> Doing a git bisect suggested that this is the culprit:
>
> commit cec0707e40ae25794b5a2de7b7f03c51961f80d9
> Author: Jens Axboe <jens....@oracle.com>
> Date: Tue Jan 13 15:28:32 2009 +0100
>
> block: silently error an unsupported barrier bio

Yes, that patch is broken.
the new code it introduces should probably go in the top of
__make_request rather than just before the call to ->make_request_fn.
make_request_fn functions other than __make_request generally
don't use ->next_ordered at all.

Jens? Would you agree?

NeilBrown

[Ken Milmore]

I thought I had better bump my previous post as this regression is still
present in 2.6.29.5.

To recap, commit cec0707e40ae25794b5a2de7b7f03c51961f80d9 has broken
write barriers on md raid1 block devices in 2.6.29 and later kernels.
Reversing this commit appears to fix the problem.

Please let me know if I'm harassing the wrong maintainers here!

-Ken.

[Neil Brown]

On Sunday June 21, ken.m...@googlemail.com wrote:
> I thought I had better bump my previous post as this regression is still
> present in 2.6.29.5.

Thanks Ken.

>
> To recap, commit cec0707e40ae25794b5a2de7b7f03c51961f80d9 has broken
> write barriers on md raid1 block devices in 2.6.29 and later kernels.
> Reversing this commit appears to fix the problem.
>
> Please let me know if I'm harassing the wrong maintainers here!

Jens,
have you had a chance to look at this?

I think the following patch is appropriate and should go in to
-stable.

Thanks,
NeilBrown

From addd8b129835a63d6df9a38eae20caaa15de5988 Mon Sep 17 00:00:00 2001
From: NeilBrown <ne...@suse.de>
Date: Wed, 24 Jun 2009 13:39:15 +1000
Subject: [PATCH] Restore barrier support for md and probably other virtual devices.

The next_ordered flag is only meaningful for devices that use __make_request.
So move the test against next_ordered out of generic code and in to
__make_request

Since this test was added, barriers have not worked on md, and (I
think) dm and similar devices that don't use __make_request and so
don't bother to set next_ordered.

Cc: sta...@kernel.org
Cc: Ken Milmore <ken.m...@googlemail.com>
Signed-off-by: NeilBrown <ne...@suse.de>
---
block/blk-core.c | 10 +++++-----
1 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/block/blk-core.c b/block/blk-core.c
index b06cf5c..fc221aa 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -1172,6 +1172,11 @@ static int __make_request(struct request_queue *q, struct bio *bio)
const int unplug = bio_unplug(bio);
int rw_flags;

+ if (bio_barrier(bio) && bio_has_data(bio) &&
+ (q->next_ordered == QUEUE_ORDERED_NONE)) {
+ bio_endio(bio, -EOPNOTSUPP)
+ return 0;
+ }
/*
* low level driver can indicate that it wants pages above a
* certain limit bounced to low memory (ie for highmem, or even
@@ -1472,11 +1477,6 @@ static inline void __generic_make_request(struct bio *bio)
err = -EOPNOTSUPP;
goto end_io;
}
- if (bio_barrier(bio) && bio_has_data(bio) &&
- (q->next_ordered == QUEUE_ORDERED_NONE)) {
- err = -EOPNOTSUPP;
- goto end_io;
- }

ret = q->make_request_fn(q, bio);
} while (ret);
--
1.6.3.1

[Alasdair G Kergon]

On Wed, Jun 24, 2009 at 01:43:35PM +1000, Neil Brown wrote:
> Since this test was added, barriers have not worked on md, and (I
> think) dm and similar devices that don't use __make_request and so
> don't bother to set next_ordered.

In dm, we added this:

+ blk_queue_ordered(md->queue, QUEUE_ORDERED_DRAIN, NULL);

Alasdair

[NeilBrown]

On Wed, June 24, 2009 5:09 pm, Alasdair G Kergon wrote:
> On Wed, Jun 24, 2009 at 01:43:35PM +1000, Neil Brown wrote:
>> Since this test was added, barriers have not worked on md, and (I
>> think) dm and similar devices that don't use __make_request and so
>> don't bother to set next_ordered.
>
> In dm, we added this:
>
> + blk_queue_ordered(md->queue, QUEUE_ORDERED_DRAIN, NULL);
>

I obviously didn't grep for the right thing.

This will obviously work, but I can't help thinking it is in the wrong
place. I don't even have a queue in md/raid1, so making a statement
about how it is ordered just doesn't make sense.

Jens?

NeilBrown

[Jens Axboe]

On Wed, Jun 24 2009, Neil Brown wrote:
> On Sunday June 21, ken.m...@googlemail.com wrote:
> > I thought I had better bump my previous post as this regression is still
> > present in 2.6.29.5.
>
> Thanks Ken.
>
> >
> > To recap, commit cec0707e40ae25794b5a2de7b7f03c51961f80d9 has broken
> > write barriers on md raid1 block devices in 2.6.29 and later kernels.
> > Reversing this commit appears to fix the problem.
> >
> > Please let me know if I'm harassing the wrong maintainers here!
>
> Jens,
> have you had a chance to look at this?

Yeah, I think it's the right way to go. I'll queue it up for .31 and we
should put in in -stable as well.

--
Jens Axboe

[Neil Brown]

On Wednesday June 24, jens....@oracle.com wrote:
> On Wed, Jun 24 2009, Neil Brown wrote:
> > On Sunday June 21, ken.m...@googlemail.com wrote:
> > > I thought I had better bump my previous post as this regression is still
> > > present in 2.6.29.5.
> >
> > Thanks Ken.
> >
> > >
> > > To recap, commit cec0707e40ae25794b5a2de7b7f03c51961f80d9 has broken
> > > write barriers on md raid1 block devices in 2.6.29 and later kernels.
> > > Reversing this commit appears to fix the problem.
> > >
> > > Please let me know if I'm harassing the wrong maintainers here!
> >
> > Jens,
> > have you had a chance to look at this?
>
> Yeah, I think it's the right way to go. I'll queue it up for .31 and we
> should put in in -stable as well.

Great, thanks.

NeilBrown