Any known issues with using fstrim in test mode?

[Andy Holt]

Hello all.

Continuing my leisurely-paced s3backer experiments, I'm keen to have
the minimum blocks stored in S3. Starting with test mode (no S3, just
a local subdirectory full of 'blocks'), I have been using fstrim, on
upper file systems formatted as ext4, xfs and ZFS (which now supports
trim). I'm seeing pretty much instance file system corruption on all
three if I run fstrim though.

Before going into detail, is this a function that is expected to work,
when in test mode?

Thanks, Andy

[Archie Cobbs]

Hmm.. it "should work"...

Are you being careful to let s3backer shutdown gracefully and exit before remounting? Note there is a bug in FUSE whereby umount will return before the unmount operation is complete. Here's an example of this affecting s3backer.

Can you create the simplest possible test scenario/recipe that demonstrates the problem?

-Archie

[Andy Holt]

Thanks Archie.

I thought it would work too. ;-)

I'll provide some detailed steps later, but in advance of that, I'll
point out that I am not unmounting anything. The fstrim tool looks to
be designed to be run on a mounted filesystem, and that's how I'm
using it, on file systems atop s3backer running in test mode. I
wonder if maybe you thought I meant I was running fstrim on the
underlying file system containing the subdirectory that I have told
s3backer (in test mode) to locally store its blocks in.

Andy

> --
> You received this message because you are subscribed to the Google Groups "s3backer-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to s3backer-deve...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/s3backer-devel/5869d942-51dc-4aaa-8b7c-dbc93ebaca55%40googlegroups.com.

[Archie Cobbs]

OK you're initializing the filesystem and then running fstrim on it all in one go?

In a simple test I was not able to reproduce this problem. Here's what I did:

In window #1:

$ mkdir mnt1 mnt2 testdir

$ sudo ./s3backer -f --debug --size=100M --blockSize=1M --test testdir ./mnt1

In window #2:

$ sudo mke2fs -t ext4 -E nodiscard -F ./mnt1/file

mke2fs 1.43.8 (1-Jan-2018)
Creating filesystem with 102400 1k blocks and 25688 inodes
Filesystem UUID: 3cc582b8-2e1b-4851-afbc-3f6700ca4ae6
Superblock backups stored on blocks:
    8193, 24577, 40961, 57345, 73729

Allocating group tables: done                           
Writing inode tables: done                           
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done
$ sudo mount -o loop mnt1/file mnt2

$ ls mnt2/
lost+found/
$ cp /etc/services mnt2/

$ ls mnt2/
lost+found/  services
$ sudo fstrim -v ./mnt2
./mnt2: 90.5 MiB (94889984 bytes) trimmed
$ diff mnt2/services /etc/services
$ sudo umount mnt2
$ sudo mount -o loop mnt1/file mnt2
$ diff mnt2/services /etc/services
$ sudo umount mnt2

I'll be interested to see how you're getting it to fail.

-Archie

On Monday, July 22, 2019 at 6:20:03 AM UTC-5, Andy Holt wrote:

Thanks Archie.

I thought it would work too.  ;-)

I'll provide some detailed steps later, but in advance of that, I'll
point out that I am not unmounting anything.  The fstrim tool looks to
be designed to be run on a mounted filesystem, and that's how I'm
using it, on file systems atop s3backer running in test mode.  I
wonder if maybe you thought I meant I was running fstrim on the
underlying file system containing the subdirectory that I have told
s3backer (in test mode) to locally store its blocks in.

Andy

On Mon, 22 Jul 2019 at 00:29, Archie Cobbs wrote:
> Hmm.. it "should work"...
>
> Are you being careful to let s3backer shutdown gracefully and exit before remounting? Note there is a bug in FUSE whereby umount will return before the unmount operation is complete. Here's an example of this affecting s3backer.
>
> Can you create the simplest possible test scenario/recipe that demonstrates the problem?
>

[Andy Holt]

Hello Archie.

My test is pretty much the same as yours. The key appears to be
volume size. If I too use a 100MiB volume, fstrim is reliable.
Increasing the volume size brings on the issue. Here are my test
commands, with a 987MiB volume:

# window 1
cd /mnt
mkdir s3backer-test s3backer-mnt fs-mnt
s3backer -f --debug --test --listBlocks --size=987M --blockSize=1m
--blockCacheWriteDelay=5000 --blockCacheSize=300 --filename=s3bbd
/mnt/s3backer-test /mnt/s3backer-mnt

# window 2
mkfs.xfs /mnt/s3backer-mnt/s3bbd
mount -o loop /mnt/s3backer-mnt/s3bbd /mnt/fs-mnt
df -h /mnt/fs-mnt
fstrim -v /mnt/fs-mnt
sleep 5
umount /mnt/fs-mnt

You should see an error like this from the fstrim:

fstrim: /mnt/fs-mnt: FITRIM ioctl failed: Input/output error

There will also be a few 'wrong MD5 checksum' errors in s3backer's
debug output, and in /var/log/syslog:

Jul 24 19:27:50 vagrant kernel: [170899.075702] XFS (loop0): Mounting
V5 Filesystem
Jul 24 19:27:50 vagrant kernel: [170899.094045] XFS (loop0): Ending clean mount
Jul 24 19:27:52 vagrant kernel: [170900.586482] print_req_error: I/O
error, dev loop0, sector 505416
Jul 24 19:27:53 vagrant kernel: [170901.908587] print_req_error: I/O
error, dev loop0, sector 1017600
Jul 24 19:27:53 vagrant kernel: [170902.020280] print_req_error: I/O
error, dev loop0, sector 1516104
Jul 24 19:27:58 vagrant kernel: [170907.033558] XFS (loop0):
Unmounting Filesystem
Jul 24 19:27:58 vagrant kernel: [170907.135379] loop: Write error at
byte offset 517493760, length 1024.
Jul 24 19:27:58 vagrant kernel: [170907.136451] print_req_error: I/O
error, dev loop0, sector 1010730
Jul 24 19:27:58 vagrant kernel: [170907.137463] print_req_error: I/O
error, dev loop0, sector 1010730
Jul 24 19:27:58 vagrant kernel: [170907.138615] XFS (loop0): metadata
I/O error in "xlog_iodone" at daddr 0xf6c2a len 64 error 5
Jul 24 19:27:58 vagrant kernel: [170907.139654] XFS (loop0):
xfs_do_force_shutdown(0x2) called from line 1250 of file
/build/linux-hwe-mCxn_j/linux-hwe-4.18.0/fs/xfs/xfs_log.c. Return
address = 00000000f04aabce
Jul 24 19:27:58 vagrant kernel: [170907.139700] XFS (loop0): Log I/O
Error Detected. Shutting down filesystem
Jul 24 19:27:58 vagrant kernel: [170907.140777] XFS (loop0): Please
umount the filesystem and rectify the problem(s)
Jul 24 19:27:58 vagrant kernel: [170907.142738] XFS (loop0): Unable to
update superblock counters. Freespace may not be correct on next
mount.

My test system is Ubuntu 18.04 with the HWE kernel (4.18.0-25), with
whatever packages are recommended on the s3backer wiki for compiling
the code, from the standard repos.

Many thanks, Andy

[Archie Cobbs]

Hi Andy,

Congrats & thanks - you found a bug in the test module :)

Should be fixed by commit b188b91.

-Archie

[Andy Holt]

Ha, excellent! :-) Thanks for looking at it.

Should I open a GitHub issue to link to it ?

Do I imply from your comment that this is only going to have affected
trim when in test mode ?

Thanks again, Andy

> --
> You received this message because you are subscribed to the Google Groups "s3backer-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to s3backer-deve...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/s3backer-devel/c1addc7f-635d-4722-a3ed-b3a985ee11f4%40googlegroups.com.

[Archie Cobbs]

Hi Andy,

On Monday, July 29, 2019 at 6:27:58 AM UTC-5, Andy Holt wrote:

Should I open a GitHub issue to link to it ?

No need really.

 

Do I imply from your comment that this is only going to have affected
trim when in test mode ?

Correct - this only affects test mode, and also (I think) this would only ever happen when the block cache is enabled as well.

-Archie