All volumes result in block checksum errors after upgrading from 18.2.7 to 19.2.5

[Kevin Thompson]

I tried upgrading from 18.2.7 to 19.2.5 today.

After upgrading, every single time stored tries to load a volume, it results in a crc checksum error while reading the first block in the volume:

06-Feb 16:37 HomeFile JobId 8110: Error: stored/block.cc:350 Volume data error at 0:0!
Block checksum mismatch in block=0 len=192: calc=f625ccaf blk=64bbb13a
06-Feb 16:37 HomeFile JobId 8110: Warning: Volume "Incr-0012" not on device "FileStorage" (/space/bareos).
06-Feb 16:37 HomeFile JobId 8110: Marking Volume "Incr-0012" in Error in Catalog.

06-Feb 16:43 HomeFile JobId 8111: Error: stored/block.cc:350 Volume data error at 0:0!
Block checksum mismatch in block=0 len=208: calc=8dc2aa1c blk=466e7c3c
06-Feb 16:37 angst-fd JobId 8111: ACL support is enabled
06-Feb 16:43 HomeFile JobId 8111: Warning: Volume "Full-0005" not on device "FileStorage" (/space/bareos).
06-Feb 16:43 HomeFile JobId 8111: Marking Volume "Full-0005" in Error in Catalog.

I suspect the files are fine:

- The files are on a raidz1 3-disk ZFS pool, and ZFS reports no sha checksum errors.

- The volumes were working just fine this morning before the update.

- The drives are in good health and there are no IO errors reported by the OS.

I noticed that 6 months ago, you completely replaced the CRC implementation used:

https://github.com/bareos/bareos/commit/838aef14bddc69241221900ba962690ac9e26203#diff-5d8d24f7d7fbd164fc11057003dd30c5

Any chance you accidentally installed a CRC implementation that gives different answers than the historical implementation? I would expect that you tested this or that someone else would've run in to this, but it's a suspicious coincidence.

For what it's worth, Bareos is able to plow ahead and save the backup to a fresh volume after recycling.

I'm running on FreeBSD 12.1 on both the client and server machine.

     FreeBSD masheen.redacted.com 12.1-STABLE FreeBSD 12.1-STABLE r357422 GENERIC  amd64

--Kevin Thompson

[Kevin Thompson]

With 19.2.5 installed, `bls` locally on the storage daemon would fail as indicated above. I downgraded to 18.2.5 and bls works again.

I think this confirms the CRC algorithm in 19.2.5 is broken.

[Andreas Rogge]

Hi Everybody,

is anyone else running into this issue?

Best Regards,
Andreas

--
Andreas Rogge andrea...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221-630693-86
http://www.bareos.com

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: S. Dühr, M. Außendorf, J. Steffens, Philipp Storz

[Andreas Rogge]

Am 07.02.20 um 07:52 schrieb Andreas Rogge:

> is anyone else running into this issue?

let me rephrase my question:
Can anyone reproduce this issue on anything besides FreeBSD?

Thank you!

[Andreas Rogge]

Hello Everybody,

we have identified a critical problem[1] in FreeBSD's SD.
Please do not upgrade a FreeBSD SD to anything before 19.2.6 (which will
include a fix) or a nightly-build until PR #412 (which fixes the
problem) has been merged.

Best Regards,
Andreas

[1] https://bugs.bareos.org/view.php?id=1180