base jobs not working properly in Always Incremental scheme
Nils Jakobson
I have a bunch of similar windows machines that I need to back up. Naturally I would like to have some deduplication to avoid backing up same files again and again. So I backed up clean empty Windows server as a base job. Now when I run a full backup (the first backup is going to be a full backup automatically) then in the log it clearly says that it is using base jobs and how many files from that base job were used for deduplication (excerpt from log):
----8<-------8<-------8<-------8<-------8<-------8<---
2019-05-05 13:51:14 pool-director JobId 2384: Using BaseJobId(s): 1
2019-05-05 15:27:06 pool-director JobId 2384: Bareos pool-director 19.1.2.1551874020.6df58d68c (01Feb19):
JobId: 2384
Job: soldevqa09.2019-05-04_21.00.00_51
Backup Level: Full
Client: "soldevqa09" 17.1.3 (01Mar17) Microsoft Windows Server 2008 R2 Standard Edition Service Pack 1 (build 7601), 64-bit,Cross-compile,Win64
FileSet: "Windows All Drives" 2018-12-13 19:20:05
Pool: "AI_Consolidated" (From Job FullPool override)
Catalog: "MyCatalog" (From Client resource)
Storage: "Consolidated01" (From Pool resource)
Elapsed time: 1 hour 35 mins 51 secs
Priority: 40
FD Bytes Written: 38,501,279,442 (38.50 GB)
SD Bytes Written: 38,610,804,604 (38.61 GB)
Rate: 6694.7 KB/s
Software Compression: 58.2 % (lzo)
Base files/Used files: 101285/81362 (80.33%)
VSS: yes
Encryption: no
Accurate: yes
----8<-------8<-------8<-------8<-------8<-------8<---
Now, the very next backup in Always Incremental scheme does not care about base jobs anymore and backs up all the duplicate stuff that was not backed up at full backup because it already had these files in Base job. I just noticed that because next daily incremental after Full backup always is significantly larger than subsequent daily incrementals. Take look at the query output:
----8<-------8<-------8<-------8<-------8<-------8<---
+-------+------------+-------+---------------------+----------+--------+---------+
| jobid | client | level | starttime | jobfiles | gb | volumes |
+-------+------------+-------+---------------------+----------+--------+---------+
| 32 | soldevqa09 | F | 2018-12-15 16:35:35 | 505,133 | 35.123 | 53 |
| 396 | soldevqa09 | I | 2018-12-27 06:56:09 | 83,152 | 5.654 | 12 |
| 220 | soldevqa09 | I | 2018-12-28 06:24:35 | 423 | 0.325 | 1 |
| 236 | soldevqa09 | I | 2018-12-29 06:40:43 | 423 | 0.325 | 2 |
| 263 | soldevqa09 | I | 2019-01-01 12:31:59 | 637 | 0.329 | 1 |
| 287 | soldevqa09 | I | 2019-01-04 03:25:51 | 1,068 | 0.386 | 1 |
| 312 | soldevqa09 | I | 2019-01-05 01:32:47 | 493 | 0.327 | 1 |
| 337 | soldevqa09 | I | 2019-01-05 10:33:03 | 326 | 0.298 | 1 |
| 365 | soldevqa09 | I | 2019-01-05 19:19:31 | 298 | 0.305 | 1 |
| 1,401 | soldevqa09 | F | 2019-02-26 08:17:56 | 508,974 | 40.382 | 81 |
| 2,392 | soldevqa09 | I | 2019-04-25 03:44:36 | 24,450 | 7.039 | 15 |
| 2,216 | soldevqa09 | I | 2019-04-26 03:40:34 | 545 | 0.197 | 1 |
| 2,239 | soldevqa09 | I | 2019-04-27 03:44:02 | 722 | 0.197 | 3 |
| 2,265 | soldevqa09 | I | 2019-04-30 04:07:52 | 1,125 | 0.200 | 1 |
| 2,289 | soldevqa09 | I | 2019-05-01 04:06:18 | 2,307 | 0.198 | 1 |
| 2,313 | soldevqa09 | I | 2019-05-02 04:17:25 | 583 | 0.275 | 1 |
| 2,337 | soldevqa09 | I | 2019-05-03 04:23:57 | 4,215 | 0.227 | 1 |
| 2,361 | soldevqa09 | I | 2019-05-04 04:27:54 | 2,061 | 0.200 | 1 |
| 2,384 | soldevqa09 | F | 2019-05-05 13:51:14 | 510,473 | 35.857 | 54 |
| 2,410 | soldevqa09 | I | 2019-05-07 04:53:22 | 82,097 | 5.666 | 12 |
+-------+------------+-------+---------------------+----------+--------+---------+
----8<-------8<-------8<-------8<-------8<-------8<---
Is this a bug or Always Incremental has never been designed to play well with Base jobs or I am doing something wrong here?
Thanks.
Neal
Oleg Volkov
Welcome to the club !
AlwaysInc is definitely buggy even before using de-duplication base job. There are a lot of open issues similar to yours (example https://bugs.bareos.org/view.php?id=907 ) I have similar issue in this thread: https://groups.google.com/forum/#!topic/bareos-users/PMuqXDqUAWQ