Sudden huge incrementals in combination with Bareos Always Incrementals

[Patrick Hasler]

We noticed a quiet urgent issue on some of our larger systems where we do backup for with Bareos. Suddenly every new incremental backup is huge and also ruffly has the same size every time (See screenshots below).

Usually I would suspect something changing on the system (For example a DB backup dump cronjob) but as this affects multiple completely unrelated systems, this can be ruled out. There is no indication of any kind of large file changes on all the affected hosts, so the question is: What is Bareos actually backing up here?

If I go to the restore section to check what files were actually backed up by the job, the "File selection" section just stays empty, so we don't have any idea what's in the backups. Due to there huge size we also cannot just restore them to the client to check their content and as we use encryption we cannot simply extract the data from the volumes on the Storage Daemon's side.

There was no recent Bareos software upgrade, so I don't think that this is a new software bug. Also the affected clients use different versions of the File Daemon (24.0.7, 25.0.2), while the Director and Storage Daemon both use 25.0.2.

What could be an explanation for this kind of behavior? I mentioned Always Incremental because I'm pretty sure that the problem is somehow related to it as the new huge incrementals started showing up after the most recent VirtualFull. But this is not the first VirtualFull, so I'm not sure why this behavior suddenly started.

[Bruno Friedmann (bruno-at-bareos)]

You can see what files have been backup'ed by using 

bconsole 

list files jobid=12345

[Patrick Hasler]

Hi Bruno,

Thank you for the quick reply! The list is actually empty:
*list files jobid=93451
*

I tried it for mutiple jobids. For older job with the regular file size the command is working as expected and I get a list of files.

[Bruno Friedmann (bruno-at-bareos)]

You may want to check what the joblog contain. 

All my AI job are showing the files backup'ed when there's some, and also all my volumes show me jobid stored on.

Are you sure you didn't get caught by the fact that after consolidation, history is rewritten, and older jobid get new files in due to the consolidation ?

[Patrick Hasler]

I've attached the job log for one of these jobs. Nothing unusual from what I can tell.

>Are you sure you didn't get caught by the fact that after consolidation, history is rewritten, and older jobid get new files in due to the consolidation?

These are not consolidation jobs, but the regular running backup jobs. Consolidation jobs would start with "Start Virtual Backup" in the joblog. This is also visible by the timestamps in the screenshots. Consolidation jobs keep their original start/end time.

[Bruno Friedmann (bruno-at-bareos)]

I hope you noticed that your database is using the wrong encoding and should be fixed as soon as possible

> Warning: Encoding error for database "bareos". Wanted SQL_ASCII, got UTF8

Try to follow our documentation when you initialize the catalog.

Due to the use of folder/file in volume names maybe you are required to quotes that when asking list files from volume and the webui can't ?

I'm not yet sure how reliable is a volume name like 

> Created new Volume "Customer-A/AI_Inc_8263" in catalog.

Is that one file in /mnt/bareos/backup or dir/file ?

> Consolidation jobs keep their original start/end time.

You mean consolidated jobs ;-) the consolidation job has its own start/end time ...

if the incremental has everyday such number of changes you might want to check them with estimate listing 

[Patrick Hasler]

> I'm not yet sure how reliable is a volume name like 

>> Created new Volume "Customer-A/AI_Inc_8263" in catalog.

> Is that one file in /mnt/bareos/backup or dir/file ?

It's an actual directory structure. In the end this will result in a sub-directory in /mnt/bareos/backup with the customer name ("Customer-A) and then the Bareos volume files in it. In this example AI_Inc_8263 is a file in the directory /mnt/bareos/backup/Customer-A.

This has worked reliably over years and was the suggested setup by Andreas Rogge. For our use-case this is very helpful because Bareos does not have a feature for "grouping" hosts together and we need to be able to tell how much storage a customer is using.

I would be very surprised if the issue is on the storage's side. The backups are properly written and the volumes have the exact file size that the WebUI and job log are reporting as well. It's just not clear what the actual content is and why they are so huge out of nowhere.

For now I've started a new consolidate job which resulted in a successful virtual full backup for some of the affected hosts. So my hope would be that the next incremental which will trigger tonight will be of regular size. 

If that would not be the case, would it be an option to have a look at the setup together tomorrow? If the problem persist we would need to find a solution as soon as possible, because we are now in a situation where we cannot trust that Bareos is actually doing proper backups of our machines. It refuses to give us the content of what's being backed up and it's filling up our storage endpoints with huge data with unknown content.

In any case, I'll let you guys know tomorrow what the status is.

> if the incremental has everyday such number of changes you might want to check them with estimate listing 

The estimate command (estimate level=incremental job=Job_Customer-A_Daily_AlwaysIncremental_client-01 accurate=yes listing) gives me a proper file list. If `level=incremental` is specified, should it only print files that have changed and would be included in the next incremental job, or will it still give a full list of all files included in the related FileSet to that job? Because the output looks like it gives us the full file tree of that system.

Here are the command outputs:

*estimate level=incremental job=Job_Customer-A_Daily_AlwaysIncremental_client-01 accurate=yes listing

... (huge list of files)

2000 OK estimate files=1,523,195 bytes=2,095,857,085,860

[Bruno Friedmann (bruno-at-bareos)]

yes estimate level=incremental will only show what will be saved.

for a given jobid you can also ask directly the database

in bconsole

.sql query="select p.path || f.name from file f left join path p on p.pathid=f.pathid where jobid=12345;"

That should return something normally.

[Patrick Hasler]

OK, we still have the same issue here. The backup jobs are still running but the volumes are already in a much larger size than they should be.

I verified the sql query as well. Here the list is also empty. But interestingly this behavior is only present for one of the affected clients. For another one where the incrementals are also way too large, I'm able to get a list of files.

For this client, I copied the list to the client and calculated the total file size, but it's not even close to the size of the related Bareos volume. So Bareos is definitely doing something wrong here. Note that we only use volumes once for a single job so the size can not be related to other jobs.

One guess I had was that Bareos is somehow not keeping track of what's in the former incrementals and is including the same files in each incremental again and again. But the incrementals are still smaller than the full system, so Bareos is not backing up the full file system each time.

The biggest affected system is around 8 TB large, but the backups are always around 1.15 TB, with the amount of files always being nearly identical with every job (805066, 811411, 810618).

How likely is it that the database encoding would cause these problems?

Approximately a month ago we shrank the Catalog database (according to https://docs.bareos.org/TasksAndConcepts/CatalogMaintenance.html#compacting-your-postgresql-database) because the PostgreSQL Autovacuum was not properly configured and the DB just kept growing more and more.

This lead to the warning regarding the database encoding. But we haven't seen any issues since then. The weird incremental growth started last week.

I'm currently waiting for one long-running consolidate job to finish but once that is done, I would dump the database and recreate it according to https://docs.bareos.org/IntroductionAndTutorial/InstallingBareos.html#id7 before adding the dump again.

It looks like the consolidate job will keep running for at least another hour before I can "fix" the DB. If we could have a look at the system together in the meantime, that would be much appreciated.