Backup stops when the backup file reaches 3Gb

[Mike Keehan]

Hi,

I'm using Qubes Backup to save some of my qubes into another VM.
The backup VM has 18 Gb of storage available, but whenever the
backup file reaches 3Gb, the backup process just hangs.

No CPU usage, no error messages, just stops. The backup window
shows 40% complete, but never moves any further (different % for
different combinations of qubes in the backup list).

After waiting a considerable time (well, 5-10 minutes), hitting
Cancel in the backup window does cancel it. The rest of the
system is continuing to work without problem. Happens every
time I try to use Qubes backup.

The Qubes Disk Space widget shows less than 50% disk used overall,
the backupVM shows only 18% disk used when the 3Gb file has been
saved.

I'm stumped.

Mike.

[js...@bitmessage.ch]

Mike Keehan:

Hi,

You may have to wait longer than 5-10 minutes. I experience something
similar when doing a full backup, except it's worse because i'm backing
up like 2.5TB. It appears to hang for several hours at a time (and this
happens more than once), but it does eventually make visible progress
again. The whole process takes over 24 hours. This is why i do full
backups very infrequently.

For you it shouldn't take nearly as long because it's a lot less data,
but the progress appearing to hang for a while seems to be normal.

I'm using 3.2 tho, and i know they made changes to the backup mechanism
under the hood in 4.0, so i'm not sure if this issue still applies in 4.0.

--
Jackie

[Chris Laprise]

Marek,

Isn't this the null bytes bug in GNU tar?

https://groups.google.com/d/msgid/qubes-users/f4d997d5-7191-06d0-e7bb-ef42745a7db5%40posteo.net

It would be a good idea to update this in dom0. My own backup tool uses
GNU tar as well.

--

Chris Laprise, tas...@posteo.net
https://github.com/tasket
https://twitter.com/ttaskett
PGP: BEE2 20C5 356E 764A 73EB 4AB3 1DC4 D106 F07F 1886

[unman]

It seems a little early to judge.

Mike - it looks from your comment as if you have been trying with
subsets of the qubes? Can you confirm if these are running or stopped.

Like jsnow, I'm regularly able to backup far more than 3G without issue,
so it looks as if there's something particular about this scenario.
It would be helpful if you could confirm the issue when all qubes you
are backing up are stopped.
Then try bisecting the qubes backup group - keep bisecting if you hit
the problem again until you either find the problematic qubes or have
backed them all up.

[Chris Laprise]

Oops, I pasted the wrong link. Here is the correct one:

https://utcc.utoronto.ca/~cks/space/blog/sysadmin/TarFindingTruncateBug

The symptoms sound remarkably the same... potentially hours-long delays
with no adverse effect on data. Trigger condition is not about size, but
due to corner cases that involve backing up nulls.

[Mike Keehan]

Thanks to all of you !!

I've been out today, but will have a look at tracking down the culprit
tomorrow:)

Mike.

[Mike Keehan]

On Thu, 24 Jan 2019 11:27:00 -0500
Chris Laprise <tas...@posteo.net> wrote:

I left a backup running before I left home this morning - it was stuck
at 1.8Gb then (confirming that size is not the issue), and it was still
at 1.8Gb when I got home this evening. That was about 8 hours or so.

I have a recollection from my dim and distant past, that I had a similar
stuck tar issue when it could not handle fifos. Not Linux in those
days. There is no mention of fifos (or sockets etc.) in the tar
manpage, so I don't know if gnu tar is supposed to handle them or not.
I had to use cpio to backup systems which contained those things.

Anyway, I'll investigate further tomorrow. You have given me a number
of things to look into. Thanks,

Mike.

[Mike Keehan]

OK, progress:)

I can backup a list of stopped qubes, running out to a 10Gb file without
any issues. They all verify OK too. However, backing up running qubes
exhibits the problem.

Starting up one of these qubes and trying to backup causes the progress
indicator to stop at some point, BUT, the data is still being backed up
to disk, and continues flowing for some time. When the data flow stops,
then I have to cancel the backup operation. However, verifying the
backup works OK - file size reported is correct, and the verify process
finishes successfully. I haven't tried to actually restore one of
these yet.

I tried recording the process status of both dom0 and the backup vm
during the backups, but could not see any particular process dying when
the progress updater stopped moving. The tar processes stopped in dom0
but I guess that was because they had finished creating the private.img
files.

So it looks like backing up a live VM causes the backup process to fail
at some point, but not before the data is actually backed up.

It doesn't look like a tar null issue as the data does get to disk OK.
It's just the control of the backup process getting out of step
somewhere.

Mike.

[Mike Keehan]

I'm trying to debug the backup halting part way when backing up
a running VM. (I have restored one of these backups successfully!)

The backup.py script shows a debug log can be produced, but I can't
find it - do I have to turn it on somehow?

Mike.