Jobs not consolidating as expected

[Brock Palen]

I keep running into a few jobs that are not consolidating saying “not enough invrementals” but there are many many incrementals.
This is a problem because eventually the file entries from the full is pruned and now I have an incremental running that is downloading almost every file. Obviously this is a problem. Not sure why some jobs are doing this and not all.

This is version 19.2.7 on Ubuntu

Job {
Name = "myth-mnt"
JobDefs = "DefaultJobAI"
Schedule = "ServerCycle"
FileSet = "myth-mnt"
Full Backup Pool = Offsite # send fulls right to tape
Spool Data = yes
}

JobDefs {
Name = "DefaultJobAI"
Type = Backup
Level = Incremental
Client = myth-fd
FileSet = "mls_std" # selftest fileset (#13)
Schedule = "WorkHours"
Storage = File
Messages = Standard
Allow Mixed Priority = yes
Priority = 4
#Allow Duplicate Jobs = no # see: https://bugs.bareos.org/view.php?id=792 Can't use this setting
Accurate = yes
Write Bootstrap = "/var/lib/bareos/%c.bsr"
Spool Data = yes

# Always Incremental Settings
AlwaysIncremental = yes
AlwaysIncrementalJobRetention = 3 weeks
Always Incremental Keep Number = 7
Always Incremental Max Full Age = 28 days
Pool = AI-Incremental
Full Backup Pool = AI-Consolidated

# For systems not always on
Reschedule on Error = yes
Reschedule Times = 5
Reschedule Interval = 1800

}


Here are the logs from the Consolidate job last night:
06-Mar 16:01 myth-dir JobId 42409: Looking at always incremental job myth-mnt
06-Mar 16:01 myth-dir JobId 42409: myth-mnt: considering jobs older than 13-Feb-2021 16:01:59 for consolidation.
06-Mar 16:01 myth-dir JobId 42409: myth-mnt: less incrementals than required, not consolidating


As you can see though the number of valid jobs below is very large and yet isn’t consolidating. Why would that be? Is it because most days the backups have no new data? (Video repository)
I have seen this though with another backup job


*list backups client=myth-fd fileset=myth-mnt
Using Catalog "myth_catalog"
+--------+----------+---------+---------------------+------+-------+----------+-----------------+-----------+
| jobid | name | client | starttime | type | level | jobfiles | jobbytes | jobstatus |
+--------+----------+---------+---------------------+------+-------+----------+-----------------+-----------+
| 36,320 | myth-mnt | myth-fd | 2020-12-05 02:09:34 | B | F | 223,267 | 313,392,913,489 | T |
| 36,391 | myth-mnt | myth-fd | 2020-12-07 06:10:45 | B | I | 0 | 0 | T |
| 36,489 | myth-mnt | myth-fd | 2020-12-09 06:48:12 | B | I | 0 | 0 | T |
| 36,522 | myth-mnt | myth-fd | 2020-12-10 02:13:42 | B | I | 0 | 0 | T |
| 36,561 | myth-mnt | myth-fd | 2020-12-11 06:47:54 | B | I | 0 | 0 | T |
| 36,597 | myth-mnt | myth-fd | 2020-12-12 07:59:01 | B | I | 0 | 0 | T |
| 36,639 | myth-mnt | myth-fd | 2020-12-13 06:14:41 | B | I | 0 | 0 | T |
| 36,668 | myth-mnt | myth-fd | 2020-12-14 04:47:16 | B | I | 0 | 0 | T |
| 36,769 | myth-mnt | myth-fd | 2020-12-16 05:44:42 | B | I | 0 | 0 | T |
| 36,799 | myth-mnt | myth-fd | 2020-12-17 06:17:29 | B | I | 0 | 0 | T |
| 36,829 | myth-mnt | myth-fd | 2020-12-18 05:52:54 | B | I | 0 | 0 | T |
| 36,869 | myth-mnt | myth-fd | 2020-12-19 02:12:57 | B | I | 0 | 0 | T |
| 36,907 | myth-mnt | myth-fd | 2020-12-20 02:22:56 | B | I | 0 | 0 | T |
| 36,936 | myth-mnt | myth-fd | 2020-12-21 02:10:50 | B | I | 0 | 0 | T |
| 37,031 | myth-mnt | myth-fd | 2020-12-23 03:51:40 | B | I | 0 | 0 | T |
| 37,074 | myth-mnt | myth-fd | 2020-12-24 02:38:25 | B | I | 0 | 0 | T |
| 37,135 | myth-mnt | myth-fd | 2020-12-25 02:27:30 | B | I | 0 | 0 | T |
| 37,175 | myth-mnt | myth-fd | 2020-12-26 02:10:22 | B | I | 0 | 0 | T |
| 37,214 | myth-mnt | myth-fd | 2020-12-27 02:42:04 | B | I | 0 | 0 | T |
| 37,241 | myth-mnt | myth-fd | 2020-12-28 02:11:34 | B | I | 0 | 0 | T |
| 37,316 | myth-mnt | myth-fd | 2020-12-30 02:10:47 | B | I | 0 | 0 | T |
| 37,370 | myth-mnt | myth-fd | 2020-12-31 07:31:39 | B | I | 0 | 0 | T |
| 37,410 | myth-mnt | myth-fd | 2021-01-01 02:14:43 | B | I | 0 | 0 | T |
| 37,446 | myth-mnt | myth-fd | 2021-01-02 02:25:32 | B | I | 0 | 0 | T |
| 37,540 | myth-mnt | myth-fd | 2021-01-04 02:10:57 | B | I | 0 | 0 | T |
| 37,586 | myth-mnt | myth-fd | 2021-01-05 02:10:47 | B | I | 0 | 0 | T |
| 37,628 | myth-mnt | myth-fd | 2021-01-06 02:10:55 | B | I | 0 | 0 | T |
| 37,682 | myth-mnt | myth-fd | 2021-01-07 02:10:40 | B | I | 0 | 0 | T |
| 37,716 | myth-mnt | myth-fd | 2021-01-08 02:11:02 | B | I | 0 | 0 | T |
| 37,755 | myth-mnt | myth-fd | 2021-01-09 03:35:11 | B | I | 0 | 0 | T |
| 37,791 | myth-mnt | myth-fd | 2021-01-10 02:49:35 | B | I | 0 | 0 | T |
| 37,822 | myth-mnt | myth-fd | 2021-01-11 02:58:00 | B | I | 0 | 0 | T |
| 37,862 | myth-mnt | myth-fd | 2021-01-12 02:10:49 | B | I | 0 | 0 | T |
| 37,904 | myth-mnt | myth-fd | 2021-01-13 02:10:47 | B | I | 0 | 0 | T |
| 37,957 | myth-mnt | myth-fd | 2021-01-14 02:10:58 | B | I | 0 | 0 | T |
| 37,995 | myth-mnt | myth-fd | 2021-01-15 06:16:23 | B | I | 0 | 0 | T |
| 38,035 | myth-mnt | myth-fd | 2021-01-16 03:08:28 | B | I | 0 | 0 | T |
| 38,067 | myth-mnt | myth-fd | 2021-01-17 08:56:54 | B | I | 0 | 0 | T |
| 38,134 | myth-mnt | myth-fd | 2021-01-18 02:08:38 | B | I | 0 | 0 | T |
| 38,167 | myth-mnt | myth-fd | 2021-01-19 05:31:28 | B | I | 0 | 0 | T |
| 38,214 | myth-mnt | myth-fd | 2021-01-20 03:44:18 | B | I | 0 | 0 | T |
| 38,254 | myth-mnt | myth-fd | 2021-01-21 03:35:20 | B | I | 0 | 0 | T |
| 38,304 | myth-mnt | myth-fd | 2021-01-22 03:51:14 | B | I | 0 | 0 | T |
| 38,364 | myth-mnt | myth-fd | 2021-01-23 02:27:06 | B | I | 0 | 0 | T |
| 38,431 | myth-mnt | myth-fd | 2021-01-24 03:29:56 | B | I | 0 | 0 | T |
| 38,462 | myth-mnt | myth-fd | 2021-01-25 02:18:53 | B | I | 0 | 0 | T |
| 38,519 | myth-mnt | myth-fd | 2021-01-26 02:11:11 | B | I | 0 | 0 | T |
| 38,553 | myth-mnt | myth-fd | 2021-01-27 02:11:34 | B | I | 0 | 0 | T |
| 38,609 | myth-mnt | myth-fd | 2021-01-28 02:23:11 | B | I | 0 | 0 | T |
| 38,653 | myth-mnt | myth-fd | 2021-01-29 02:11:26 | B | I | 0 | 0 | T |
| 38,700 | myth-mnt | myth-fd | 2021-01-30 02:12:18 | B | I | 0 | 0 | T |
| 38,802 | myth-mnt | myth-fd | 2021-02-01 02:11:30 | B | I | 0 | 0 | T |
| 38,900 | myth-mnt | myth-fd | 2021-02-02 04:06:15 | B | I | 51 | 421,596 | T |
| 38,933 | myth-mnt | myth-fd | 2021-02-03 02:11:14 | B | I | 0 | 0 | T |
| 38,977 | myth-mnt | myth-fd | 2021-02-04 02:19:01 | B | I | 0 | 0 | T |
| 39,020 | myth-mnt | myth-fd | 2021-02-05 03:51:45 | B | I | 0 | 0 | T |
| 40,653 | myth-mnt | myth-fd | 2021-02-06 03:23:35 | B | I | 0 | 0 | T |
| 40,702 | myth-mnt | myth-fd | 2021-02-07 02:28:14 | B | I | 0 | 0 | T |
| 40,754 | myth-mnt | myth-fd | 2021-02-08 02:09:55 | B | I | 0 | 0 | T |
| 40,799 | myth-mnt | myth-fd | 2021-02-09 02:09:56 | B | I | 68 | 518,864 | T |
| 41,389 | myth-mnt | myth-fd | 2021-02-10 06:05:10 | B | I | 0 | 0 | T |
| 41,455 | myth-mnt | myth-fd | 2021-02-11 02:11:43 | B | I | 0 | 0 | T |
| 41,497 | myth-mnt | myth-fd | 2021-02-12 02:12:05 | B | I | 0 | 0 | T |
| 41,542 | myth-mnt | myth-fd | 2021-02-13 02:55:16 | B | I | 0 | 0 | T |
| 41,588 | myth-mnt | myth-fd | 2021-02-14 02:18:03 | B | I | 30 | 102,838 | T |
| 41,646 | myth-mnt | myth-fd | 2021-02-15 02:12:20 | B | I | 0 | 0 | T |
| 41,685 | myth-mnt | myth-fd | 2021-02-16 02:11:36 | B | I | 39 | 416,310 | T |
| 41,722 | myth-mnt | myth-fd | 2021-02-17 07:41:52 | B | I | 0 | 0 | T |
| 41,761 | myth-mnt | myth-fd | 2021-02-18 07:02:43 | B | I | 0 | 0 | T |
| 41,795 | myth-mnt | myth-fd | 2021-02-19 06:18:10 | B | I | 0 | 0 | T |
| 41,836 | myth-mnt | myth-fd | 2021-02-20 04:19:51 | B | I | 0 | 0 | T |
| 41,870 | myth-mnt | myth-fd | 2021-02-21 02:11:03 | B | I | 0 | 0 | T |
| 41,910 | myth-mnt | myth-fd | 2021-02-22 07:50:08 | B | I | 0 | 0 | T |
| 41,947 | myth-mnt | myth-fd | 2021-02-23 03:10:37 | B | I | 65 | 519,188 | T |
| 41,977 | myth-mnt | myth-fd | 2021-02-24 02:16:04 | B | I | 0 | 0 | T |
| 42,016 | myth-mnt | myth-fd | 2021-02-25 03:15:55 | B | I | 0 | 0 | T |
| 42,051 | myth-mnt | myth-fd | 2021-02-26 07:31:42 | B | I | 0 | 0 | T |
| 42,143 | myth-mnt | myth-fd | 2021-02-28 11:07:05 | B | I | 0 | 0 | T |
| 42,168 | myth-mnt | myth-fd | 2021-03-01 02:09:16 | B | I | 0 | 0 | T |
| 42,207 | myth-mnt | myth-fd | 2021-03-02 09:35:16 | B | I | 63 | 519,439 | T |
| 42,239 | myth-mnt | myth-fd | 2021-03-03 08:27:48 | B | I | 0 | 0 | T |
| 42,279 | myth-mnt | myth-fd | 2021-03-04 02:11:16 | B | I | 0 | 0 | T |
| 42,311 | myth-mnt | myth-fd | 2021-03-05 07:10:55 | B | I | 0 | 0 | T |
| 42,356 | myth-mnt | myth-fd | 2021-03-06 08:03:55 | B | I | 0 | 0 | T |




Brock Palen
bro...@mlds-networks.com
www.mlds-networks.com
Websites, Linux, Hosting, Joomla, Consulting

[Brock Palen]

I found the issue, not sure if bug for feature or how to change the functions,
I enabled director debuging
setdebug level=99 trace=1 dir

And the issue is always incremental calculation only considers jobs that have changes, eg below in the log the job ID’s listed are only jobs that have files in the backup. All the zero entry backups are not considered.
Becuase this is a media repository ti’s only added to rarely, so it doesn’t consolidate often, but eventually the File Retention setitng for the host expires and that’s when it starts backing up again that explains why it’s an incremental, (the job record is still there).

myth-dir (1): cats/sql_get.cc:1412-42462 db_accurate_get_jobids=36320,38900,40799,41588,41685,41947,42207,42432
myth-dir (10): dird/consolidate.cc:160-42462 unlimited jobids list: 36320,38900,40799,41588,41685,41947,42207,42432.
myth-dir (10): dird/consolidate.cc:179-42462 myth-mnt: considering jobs with ClientId 1 and FilesetId 12 older than 14-Feb-2021 20:44:56 for consolidation.
myth-dir (1): cats/sql_get.cc:1412-42462 db_accurate_get_jobids=36320,38900,40799,41588
myth-dir (10): dird/consolidate.cc:183-42462 consolidate candidates: 36320,38900,40799,41588.
myth-dir (10): dird/consolidate.cc:204-42462 Incrementals found/required. (7/7).

Is there a way to tell Consolidate to consider jobs of size 0? I could drastically increases the file retention time for the media volume, but this is also causing me to have tapes hanging around with one job not purged, taking up a lot of auto loader space.

I’ll try a work around by adding a RunScript to the job so when it runs it just drops a time stamp into the media volume so every time it runs it gets at least 1 file,
This is still not the behavior I expected, I exepcted “incremental required” to mean actual jobs independed of changes found. But I can also see how someone would want the last X of actual changes. In my case I only care about a window ~ X days back in the event of failure, crypto, staff error.

If there is a setting for this (looking at the Director Job config don’t see anyhting sticking out) to change the behavior let me know. Otherwise I’ll try the hunch of a work around.

Brock Palen
bro...@mlds-networks.com
www.mlds-networks.com
Websites, Linux, Hosting, Joomla, Consulting

> --
> You received this message because you are subscribed to the Google Groups "bareos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/B2419682-90CB-4106-A35C-599C15844753%40mlds-networks.com.