Always Incremental Max Full Age
Jens Köhler
Hey guys,
I am wondering why our Always Incremental setup also consolidates the Full backup every 2 days, even though we set
Always Incremental Max Full Age = 14 days
The last Consolidation mails look like:
08-Mar 04:00 bareos-dir JobId 442: check full age: full is 19-Feb-2018 21:18:50, allowed is 22-Feb-2018 04:00:02
08-Mar 04:00 bareos-dir JobId 442: Full is older than AlwaysIncrementalMaxFullAge -> also consolidating Full jobid 434
-
10-Mar 04:00 bareos-dir JobId 452: check full age: full is 21-Feb-2018 21:18:57, allowed is 24-Feb-2018 04:00:02
10-Mar 04:00 bareos-dir JobId 452: Full is older than AlwaysIncrementalMaxFullAge -> also consolidating Full jobid 444
-
12-Mar 04:00 bareos-dir JobId 462: check full age: full is 23-Feb-2018 21:18:10, allowed is 26-Feb-2018 04:00:02
12-Mar 04:00 bareos-dir JobId 462: Full is older than AlwaysIncrementalMaxFullAge -> also consolidating Full jobid 454
-
I would have expected the Consolidate job to reset the full age to the last time of Consolidation. For example, the Consolidate job runs on 08-Mar, also consolidates Full and sets Full age to 08-Mar. The next time where Full backup is also part of Consolidation should be 22-Mar, am I wrong?
Thanks in advance
Dan
This isn't completely intuitive. My guess is that you have something like ...
Always Incremental Job Retention = 12 days
Always Incremental Max Full Age = 14 days
When it consolidated the full it only consolidated it with the oldest 2 incrementals (because it has to retain 12 incrementals), so it advanced the date of the new full by 2 days. Two days later it's ready to consolidate the full again because it is again 14 days old.
By way of example, I have the following configuration ...
Always Incremental Job Retention = 7 days
Always Incremental Max Full Age = 21 days
This actually consolidates with the full backup every 14 days. When the full is 21 days old it does the 1st full consolidation, with the oldest 14 days worth of incrementals. This advances the date of the new full backup by 14 days. So 14 days later it will again include the Full in the consolidation because it is again 21 days old.
The frequency of the Full consolidation is 'Always Incremental Max Full Age' minus 'Always Incremental Job Retention = 7 days'.
Dan
Jens Köhler
my JobDef for Always Incremental is:
Accurate = yes
Always Incremental = yes
Schedule {
Name = "AlwaysIncremental"
Run = Incremental daily at 21:00
}
Schedule {
Name = "Consolidate"
Run = Incremental daily at 04:00
}
So the frequency of Full consolidation should be 7 days as in your example.
Despite the fact that I'm kind of confused by now, I would expect Bareos to consolidate 1 old Incremental per day and include Full into consolidation once a week. But instead every 2 days it will consolidate 2 old Incrementals and the Full. The other days it says "less incrementals than required to consolidate without full, not consolidating".
Dan
> my JobDef for Always Incremental is:
> Accurate = yes
> Always Incremental = yes
> Always Incremental Job Retention = 7 days
> Always Incremental Keep Number = 14
> Always Incremental Max Full Age = 14 days
Jens -
"Always Incremental Keep Number = 14" is the culprit. That value should always be set less than or equal to "Always Incremental Job Retention", at least when you are backing up once per day. Think of "Always Incremental Keep Number" as the minimum number of Incremental backups you want to retain in the event one or more of the daily backups fails to run.
As an aside, think about why you are running consolidation every day. My inclination would be to run it once a week, or maybe once every 2 weeks. One reason that I can think of to run it more often is if you want to spread out the load of doing many client Full consolidations on the same day. In that case you can set "Max Full Consolidations" to some subset of your clients and run the consolidate more frequently.
Again by example ... if you have 12 clients and want to limit the number of Full consolidations to 3, then you can run consolidation every 4 days and each client will consolidate its full backup every 4th consolidation cycle (every 16 days). This is actually the way I'm configured because I offsite my backups every day and didn't want the network load of consolidating all of the Full backups on the same day. Note that, per my previous post, I have "Always Incremental Job Retention = 7 days" and "Always Incremental Max Full Age = 21 days", which results in eligibility for a Full consolidation every 14 days. I chose a 4 day interval on my consolidations because at the 3rd interval (12 days) the Full won't run and at the 4th interval (16 days) the Full will run. This guarantees that Bareos won't keep doing Full consolidations on the same 3 clients, leaving the other 9 clients with unlimited Incrementals.
Sorry for the unsolicited opinion there. I just haven't yet seen a good justification for performing daily Consolidations.
Dan
Jens Köhler
thanks for your detailed explanation!
I changed the Always Incremental Keep Number value to 7 and additionally limited the Consolidate schedule to "Incremental fri at 04:00". This is also a better setup for our subsequent tape backup as we do not have to write the consolidated data more than once per week.
Gaetan SLG
Dan
Attached zip contains schedules, pools, job, script, etc for 2 clients, 1 of which is the bareos server. Consolidation is run every 4th day for a maximum of 1 client. Since each of the clients is eligible for consolidation after 14 days that means in steady state each client will be consolidated on the 4th consolidation schedule (4 x 4 days = 16 days which is greater than the 14 days required). For more AI clients if you wanted to maintain the 4 day consolidation schedule and 16 day consolidation of each client, you would increase MaxFullConsolidations in the Consolidate.conf job. Alternatively you could change the consolidate schedule to run it every other day or every day. Or a combination of both to spread the load as desired.