bup reading 5TB data from NAS for saving 1% of 2TB

[karl.ri...@gmail.com]

Hi,

I'm using bup 0.29-3 in order to backup a 2TB partition with 6 million files on a cifs mount on a NAS. The first run of `bup index` and `bup save` took about a week with 6MB/s max. transfer onto the NAS/cifs mount. Now, the second only has a few KB/s transfer rate and will take months (no change after a day). After one day 1% is done and 5TB are transferred from the NAS to the laptop where I'm running bup according to `gnome-system-manager` (some other traffic happening as well, but not more than 10GB). There's constant download activity with the max. transfer rate (approx. 100MB/s) from the NAS to my laptop. Should I consider this a bug or a design issue? Any change of using bup for this task?

-Kalle

[Rob Browning]

karl.ri...@gmail.com writes:

> Hi, I'm using bup 0.29-3 in order to backup a 2TB partition with 6
> million files on a cifs mount on a NAS. The first run of `bup index`
> and `bup save` took about a week with 6MB/s max. transfer onto the
> NAS/cifs mount.

Perhaps plausible if it was averaging close to 4MB/s (and if I did the
math right)...

> Now, the second only has a few KB/s transfer rate and will take months
> (no change after a day). After one day 1% is done and 5TB are
> transferred from the NAS to the laptop where I'm running bup according
> to `gnome-system-manager` (some other traffic happening as well, but
> not more than 10GB). There's constant download activity with the
> max. transfer rate (approx. 100MB/s) from the NAS to my laptop. Should
> I consider this a bug or a design issue? Any change of using bup for
> this task?

(I wonder why it can transfer 100MB/s now, but was much slower during
the initial save...)

What command is running, index, save, or something else?

And I don't really know much about cifs if that's relevant, but in
general, what's the context, i.e. it sounds like bup is running locally,
but is the repository on a local filesystem, or on the network
filesytem, and is the data being saved on a local filesystem, or on the
remote filesystem?

And what does bup say it's doing (if anything).

--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

[Karl-Philipp Richter]

Hi Rob,

Am 27.08.19 um 08:40 schrieb Rob Browning:

> Perhaps plausible if it was averaging close to 4MB/s (and if I did the
> math right)...

I don't know about the average. I never saw it go above 6MB/s when I
took a look.

> (I wonder why it can transfer 100MB/s now, but was much slower during
> the initial save...)

100 MB/s is the transfer rate of the network filesystem. `bup` never
reached such transfer rates. It constantly reads about 20 times more
from the filesystem it transfers to than it actually transfers.

The bup directory is located on the cifs mount, so I can observe the
transfer rate in gnome-system-monitor.

> What command is running, index, save, or something else?

`bup index` takes about 10 minutes which is fine for 2TB and 6M files.
`bup save` is the troublemaker I'm describing.

> And I don't really know much about cifs if that's relevant, but in
> general, what's the context, i.e. it sounds like bup is running locally,
> but is the repository on a local filesystem, or on the network
> filesytem, and is the data being saved on a local filesystem, or on the
> remote filesystem?
>
> And what does bup say it's doing (if anything).

The `bup index` and `save` are running as processes on my laptop. The
laptop has a cifs mounted where the `bup` directory is located. The cifs
appears as mount point in the filesystem. It appears local (for `bup`
and any other software) and data written on it and read from it is
transferred over network behind the scenes. I can image many people use
a NAS as backup target device and that this is a common use case - or
unfortunately a reason not to use bup in this constallation.

The 6Mb/s transfer rate is acceptable - even though I don't see why the
throughput is so small, but a backup taking months with 0.1Mb/s is not
feasible. That being said, I'd like to provide you with means
(information, logs, tests of patches) to improve the software.

-Kalle

[Rob Browning]

Karl-Philipp Richter <karl.ri...@gmail.com> writes:

> The `bup index` and `save` are running as processes on my laptop. The
> laptop has a cifs mounted where the `bup` directory is located. The cifs
> appears as mount point in the filesystem.

Is the data being saved on the remote mount too, or is that data local
to the laptop?

[Karl-Philipp Richter]

Am 31.08.19 um 20:27 schrieb Rob Browning:

> Is the data being saved on the remote mount too, or is that data local
> to the laptop?

The data is saved into the bup directory located on the mount. The `bup
index` and `bup save` processes read from that directory and write into
it. The cifs filesystem driver takes care about system calls reading
from the remote counterpart rather than from local HDD.

[Rob Browning]

Right, but are the paths you're indexing/saving on the remote mount or
local? i.e. if you're running "bup save /foo" is /foo handled by cifs
or your local drive?

Thanks

[Karl-Philipp Richter]

Hi Rob,

Am 01.09.19 um 00:03 schrieb Rob Browning:

> Right, but are the paths you're indexing/saving on the remote mount or
> local? i.e. if you're running "bup save /foo" is /foo handled by cifs
> or your local drive?

Now, I get your question. The path passed to `bup index` and `save` is
`/` and the mountpoint including the bup directory as well as the cifs
moount point are excluded with `--exclude=/path/to/directory` together
with a few dozens other directories excluded for other reasion.

I'm doing a new backup with a fresh repository now which will not use /
as base, but approx. 10 directories which in case the issue is caused by
expensive calculation of exclusions. I'll report back in a week when the
backup is finished. Thanks for you help so far.

-Kalle

[Rob Browning]

Hmm, I'll be surprised if it's specifically related to the cost of
exclusions (detecting/excluding them should be quite cheap).

If you aren't, and the output isn't a problem somehow, might be worth
specifying -v (or -vv, though -v may be enough) to save so you can see a
bit more about what it's doing.

Also, did I mention index's --no-check-device? In some cases, that's
critical in order to prevent bup from having to re-scan some/all of the
data. i.e. it's definitely needed here for lvm snapshot mounts because
the device can change every time the snapshot is deleted/recreated.

Without --no-check-device, bup will take the changed id as a sign that
all of the content may have changed (and so it must re-compute the
hashes). No idea offhand if that's relevant to your situation, but you
can check it via stat, i.e., it's in the "Device" value here:

$ stat /tmp
File: /tmp
Size: 20480 Blocks: 40 IO Block: 4096 directory
Device: fd02h/64770d Inode: 181497 Links: 29
Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2018-12-28 13:42:50.975164141 -0600
Modify: 2019-09-02 11:56:28.119251137 -0500
Change: 2019-09-02 11:56:28.119251137 -0500
Birth: -

If that value can changes when you unmount/remount your cifs volume,
reboot, or for any other reason, then you alsmost certainly want to
specify --no-check-device.

Though you may not want to specify it for a "shared" mount point like
/mnt (commonly), i.e. one where the changing device id really might be a
signal that the device changed and the data may be different, even if
everything else matches up.

[Nix]

On 2 Sep 2019, Rob Browning outgrape:

> Karl-Philipp Richter <karl.ri...@gmail.com> writes:
>
>> Am 01.09.19 um 00:03 schrieb Rob Browning:
>>> Right, but are the paths you're indexing/saving on the remote mount or
>>> local? i.e. if you're running "bup save /foo" is /foo handled by cifs
>>> or your local drive?
>> Now, I get your question. The path passed to `bup index` and `save` is
>> `/` and the mountpoint including the bup directory as well as the cifs
>> moount point are excluded with `--exclude=/path/to/directory` together
>> with a few dozens other directories excluded for other reasion.
>>
>> I'm doing a new backup with a fresh repository now which will not use /
>> as base, but approx. 10 directories which in case the issue is caused by
>> expensive calculation of exclusions. I'll report back in a week when the
>> backup is finished. Thanks for you help so far.
>
> Hmm, I'll be surprised if it's specifically related to the cost of
> exclusions (detecting/excluding them should be quite cheap).

Agreed. I routinely run with thousands of exclusions and sometimes tens
of thousands, and there is no perceptible performance cost.

[Gabriel Filion]

For what it's worth: exclusions are only considered during the indexing
process. Once your index has been built, "bup save" relies on the index
as a list of what to backup (or avoid if it hasn't changed) and the
excluded files/directories should not be present in the index.

[Karl-Philipp Richter]

Hi,

Am 26.08.19 um 18:01 schrieb karl.ri...@gmail.com:

I did some investigation over the weeks with cifs and nfs - both after
optimizing cache and I/O performance. The best result for the initial
save of 2TB was with nfs and server-side async option enabled. It still
took 4 days. The problem with impossible daily backup remains: In order
to save 80GB new files, `bup save` read 2TB from the target filesystem
already and has a progress of 15% after 12 hours.

Is this the use case to save 2TB of data covered by bup? I suspect a
part of the software to not scale and cause the large amount unnecessary
reads as soon as the amount of data or the number of files grows.

-Kalle

[Rob Browning]

Karl-Philipp Richter <karl.ri...@gmail.com> writes:

> Is this the use case to save 2TB of data covered by bup? I suspect a
> part of the software to not scale and cause the large amount unnecessary
> reads as soon as the amount of data or the number of files grows.

I'm still not sure I have a good understanding of your arrangement, but
in case it helps reason about it, and ignoring the underlying filesystem
type, bup should only need to read the contents of files that have
"changed"[1] between index/save runs, but it will have to read the entire
contents of each of those files in order to compute the fingerprints.

So, for example if the tree being indexed/saved only had a lot of large
VM images, each of which changed in small ways between backups, then bup
would have to read all of the data on every save, even though it
(likely) ended up not actually writing much new information to the
repository.

In terms of the storage for the repository itself, bup currently relies
very heavily on mmap, and during index/save it's likely to issue a lot
of scattered reads to the idx/midx/bloom files in the repository, along
with writes for the actual data being saved, which will be appends to
the new packfile(s) it creates.

So unless something unusual is going on due to your particular
arrangment, say an issue with cifs or something (which we might be able
to improve), then if you really do have either 2TB of actual change each
time, or files that bup thinks has changed (via modification time etc.)
whose sizes add up to ~2TB, then bup will have to read that much data.

There's no real alternative until/unless we have a filesystem interface
that provides a way to find out what regions of a large file have
changed, or something similar.

Oh, and I think I mentioned, but if your arrangement needs
--no-check-device and aren't specifying it, that can also cause bup
index to have to "start from scratch" every time, which means re-reading
*all* of the data.

[1] https://github.com/bup/bup/blob/86e9359072ac5b11fe9eec17ad352ca39949be1a/lib/bup/index.py

[Karl-Philipp Richter]

Hi,

Am 26.08.19 um 18:01 schrieb karl.ri...@gmail.com:

After experimenting a lot with both cifs and NFS options and bup
versions, I got a good result with 0.29-3 and by using the cifs mount
options

user=admin,rw,vers=3,gid=1234,uid=1234,mfsymlinks,cache=loose

The initial backup of 2TB was completed after three days and even large
updates of 100GB only take about 5 hours.

Thanks for your support and advice.

-Kalle