Backup speed goes from 11MB/s to 30KB/s after a few hours with large? numbers of files.

[Andy Monkiewicz]

I am trying to get backups working with Bareos and have been having a lot of trouble with the fd just giving up doing anything with any speed after a while. The backup starts smooth at 10-20MB/s (I'm watching with bmon), then after an hour or 2, just sits at 35KB/s.

Both machines are new, and only one backup is configured. Both are running Ubuntu 14.04 and whatever the newest copy of Bareos is as of last week. They are on the same GB switch, and iperf tests show the expected available bandwidth as ~950 Mbits/sec.

The director/storage computer is running postgresql as the db backend, and if I sit and watch top as the job is running, only occasionally does bareos-sd hit 1%, the rest of the time sitting completely idle. Generally looking like this:

Tasks: 161 total, 1 running, 160 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.1 us, 0.2 sy, 0.0 ni, 99.5 id, 0.2 wa, 0.0 hi, 0.0 si, 0.0 st

The source computer has folder with the data to be backed up, which is ~130GB and ~2M files. It isn't doing anything but being a backup source.

I did notice the source computer with some i/o wait time:
Tasks: 219 total, 1 running, 217 sleeping, 0 stopped, 1 zombie
%Cpu(s): 0.1 us, 0.1 sy, 0.0 ni, 74.9 id, 25.0 wa, 0.0 hi, 0.0 si, 0.0 st

But that doesn't seem to explain why it was trucking along at 10-20MB/sec and then just dying after an hour and a half or so. I don't feel like 2 million files and 100 gigs is a lot these days? In any case it seems like a transfer rate of 30-60KB/s is just horrid performance.

The filesystem is ext3 (and is on an encrypted volume). Is there a better file system I should be using that would enhance performance? I have attribute spooling on for this job.

And now after sitting at 35K/s for an hour while writing this post, it has jumped back up to 10MB/s for 30 seconds and then dropped back to 50KB/s!

Help!

[Andy Monkiewicz]

I have switched the filesystem from ext3 to ext4 mounted with noatime and now I'm all the way up to 52KB/s. It was running strong: (Files=673,386 Bytes=73,533,094,047 Bytes/sec=18,187,755) until this point in the backup, about an hour in, then dropped speed all the way down to 45-50KB/s.

[Bruno Friedmann]

Even if the hardware is new, it doesn't mean it work as expected.
You could be out of luck and get a bad media which has pending sectors or things like that.

Try to identify you firmware hdd and check you have the latest available from the Mfg.
Run smartctl long test and check the result.
If you can dd in read mode the whole disk and also on write mode.

It also takes days but mkfs -cc can detect bad sectors


The other idea that come to my mind would be a wrongly luks encrypted but that seems improbable
to fail this init with recent tools.


--

Bruno Friedmann
Ioda-Net Sàrl www.ioda-net.ch
Bareos Partner
openSUSE Member, fsfe fellowship
GPG KEY : D5C9B751C4653227
irc: tigerfoot

[Stephan Dühr]

That's an average file size of about 100 KB, but you may have a few very
big files and and large number of very small files.

It's normal that speed goes down with a large number of small files. Also the filesystem cache is relevant here.

You could measure indepentently from Bareos how much time it takes on your system to read
all the files by using tar like this

time tar --totals -cf /dev/null /path/to/your/data

and observe iostat as you did before.

To make sure all is really read from disk, clear the cache before, by using

echo 2 > /proc/sys/vm/drop_caches

The caching for sure depends on how much RAM you have and how much of
it is available for caching.

Repeat the tar command without clearing the cache to see the effect.

Normally, nowadays relatime is used by default, which only modifies atime
on changes. So probably noatime won't make much difference.

If you don't use Data Spooling (see http://doc.bareos.org/master/html/bareos-manual-main-reference.html#DataSpooling),
make sure to use Attribute Spooling by setting

Spool Attributes = yes

See http://doc.bareos.org/master/html/bareos-manual-main-reference.html#directiveDirJobSpool%20Attributes

Otherwise data is written into the Catalog DB while the backup is running, this may slow down
the backup process. When using attribute spooling, all Catalog DB writes will happen batched
at the end of the job.

Regards

--
Stephan Dühr stepha...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221-630693-90
http://www.bareos.com

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: S. Dühr, M. Außendorf,
J. Steffens, Philipp Storz, M. v. Wieringen

[Kjetil Torgrim Homme]

Stephan Dühr <stepha...@bareos.com> writes:

> You could measure indepentently from Bareos how much time it takes on
> your system to read all the files by using tar like this
>
> time tar --totals -cf /dev/null /path/to/your/data
>
> and observe iostat as you did before.

sorry about the late response, but I want to point out that GNU tar
detects when the output file is /dev/null and does *not* read data in
this case. I believe this was an optimisation requested from the AMANDA
people :-)

the workaround is to specify /dev/zero as the output device.

--
Kjetil T. Homme
Redpill Linpro AS - Changing the game