Incremental backup do same (static) files.
vol...@gmail.com
After implementing AlwaysInc schema, I've payed attention that my incremental backup are quite large. Probably it was also true for classic Full-Diff-Inc schema or not, I am not sure. Inspecting files included in Incremental backup, I can see same static unchanged files backed up every day. I thought about MD5 hash collision, then I've replaced the signature to be SHA256. After Full backup, a next Incremental brings me almost same list of backed up files. The noatime directive also not helped.
What is strategy of selecting files to be backed up ?
If "accurate=pins5" in fileset is works in AlwaysInc mode , or its own "accurate=yes" in jobdef breaks selection ?
Also, I saw still open bug https://bugs.bareos.org/view.php?id=907 related to Windows. I see the same problem on linux.
Bareos Version is 17.2.4
My Fileset is:
FileSet {
Name = "HANA-001"
Description = "Backup local ext2,3,4 and xfs FS"
Include {
Options {
signature=SHA256
onefs=Yes
compression=GZIP
accurate=pins5
sparse=yes
noatime=yes
# keepatime=yes
}
File = "\\|sh -c \"mount | egrep 'ext[2-4]|xfs' | awk '{print \$3}' \""
Exclude Dir Containing = .nobareos
}
Exclude {
File = /var/lib/bareos
File = /var/lib/bareos/storage
File = /proc
File = /sys
File = /tmp
File = /var/tmp
File = /dev
File = /.journal
File = /.fsck
}
}
My jobdef is
JobDefs {
Accurate = yes
Always Incremental = yes
Always Incremental Job Retention = 30 days
Always Incremental Keep Number = 6
# Always Incremental Max Full Age = 31 days
Client = bareos-fd
FileSet = "SelfTest" # selftest fileset
Full Backup Pool = Full
Incremental Backup Pool = Incremental
Level = Incremental
Messages = Standard
Name = "AlwaysInc"
Next Pool= "Full"
Pool = "Incremental"
Priority = 10
Schedule = "AlwaysInc"
Storage = File
Type = Backup
}
Andreas Rogge
but could never pinpoint what exactly happens here and why.
As we cannot reproduce what is broken here, we will need somebody who
can reproduce and test in his envrionment.
Are you able and willing to do this? You might need to run patched
versions of the file-daemon and director for that.
> What is strategy of selecting files to be backed up ?
> If "accurate=pins5" in fileset is works in AlwaysInc mode , or its own "accurate=yes" in jobdef breaks selection ?
> Also, I saw still open bug https://bugs.bareos.org/view.php?id=907 related to Windows. I see the same problem on linux.
happen on every type of system.
Best Regards,
Andreas
--
Andreas Rogge andrea...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221-630693-86
http://www.bareos.com
Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: S. Dühr, M. Außendorf, J. Steffens, Philipp Storz
Oleg Volkov
Am 27.04.19 um 16:54 schrieb vol...@gmail.com:
> After implementing AlwaysInc schema, I've payed attention that my incremental backup are quite large. Probably it was also true for classic Full-Diff-Inc schema or not, I am not sure. Inspecting files included in Incremental backup, I can see same static unchanged files backed up every day. I thought about MD5 hash collision, then I've replaced the signature to be SHA256. After Full backup, a next Incremental brings me almost same list of backed up files. The noatime directive also not helped.
You may have hit a bug here. We have heard of issues like that before
but could never pinpoint what exactly happens here and why.
As we cannot reproduce what is broken here, we will need somebody who
can reproduce and test in his envrionment.
Are you able and willing to do this? You might need to run patched
versions of the file-daemon and director for that.
> What is strategy of selecting files to be backed up ?
> If "accurate=pins5" in fileset is works in AlwaysInc mode , or its own "accurate=yes" in jobdef breaks selection ?
I did not understand this. Can you rephrase it please?
> Also, I saw still open bug https://bugs.bareos.org/view.php?id=907 related to Windows. I see the same problem on linux.
While the bug has been filed for a Windows client this will probably
happen on every type of system.
Best Regards,
Andreas
--
Andreas Rogge andrea...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221-630693-86
http://www.bareos.com
Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: S. Dühr, M. Außendorf, J. Steffens, Philipp Storz
Andreas Rogge
> Give me a couple of days for that, it could be testing server for beginning.
fix it.
> > What is strategy of selecting files to be backed up ?
> > If "accurate=pins5" in fileset is works in AlwaysInc mode , or its
> own "accurate=yes" in jobdef breaks selection ?
> I did not understand this. Can you rephrase it please?
>
> My fileset definitions has "accurate" directive and my jobdef (both
> posted in original mail) has also "accurate" directive.
> Are they conflicts in some way ?
The accurate in Job/JobDefs enables and disables accurate
The accurate in the fileset's options configures how accurate decides
wether a file needs to be backed up or not.
Best Regards,
Oleg Volkov
Jörg Steffens
> Interesting, that using another server from scratch and higher version
> of director, I still see same files in AlwaysIncremental backup.
> Lets start debug the issue.
FileSet {
Name = "HANA-001"
...
signature=SHA256
...
accurate=pins5
The "5" in "pins5" means "compare the MD5 signature" while your are
storing the SHA256 signature.
I wonder if this may be the cause for your problems.
Jörg
--
Jörg Steffens joerg.s...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221 630693-91
http://www.bareos.com Fax: +49 221 630693-10
Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer:
Oleg Volkov
--
You received this message because you are subscribed to a topic in the Google Groups "bareos-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bareos-users/PMuqXDqUAWQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bareos-users...@googlegroups.com.
To post to this group, send email to bareos...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
vol...@gmail.com
# /usr/sbin/bareos-fd -d 99 -f -m > /tmp/debug 2>&1
Inspecting resulting files, found that accurate.c do correct selection.
It picks up 96 really changed files, summary about 100m.
However, the resulting job shows:
gibuy-dir JobId 22: Bareos gibuy-dir 18.2.5 (30Jan19):
Build OS: Linux-4.4.92-6.18-default redhat CentOS Linux release 7.6.1810 (Core)
JobId: 22
Job: fd-sle11sp4.2019-05-02_13.46.07_48
Backup Level: Incremental, since=2019-05-01 22:10:09
Client: "sle11sp4" 17.2.4 (21Sep17) x86_64-suse-linux-gnu,suse,SUSE Linux Enterprise Server 11 (x86_64),SLE_11_SP4,x86_64
FileSet: "HANA-001" 2019-04-30 22:52:10
Pool: "Incremental" (From command line)
Catalog: "MyCatalog" (From Client resource)
Storage: "File" (From Job resource)
Scheduled time: 02-May-2019 13:46:07
Start time: 02-May-2019 13:46:11
End time: 02-May-2019 13:50:27
Elapsed time: 4 mins 16 secs
Priority: 10
FD Files Written: 260
SD Files Written: 260
FD Bytes Written: 4,159,639,508 (4.159 GB)
SD Bytes Written: 4,159,673,113 (4.159 GB)
Rate: 16248.6 KB/s
Software Compression: 87.7 % (gzip)
VSS: no
Encryption: no
Accurate: yes
Volume name(s): Incremental-0010
Volume Session Id: 5
Volume Session Time: 1556710222
Last Volume Bytes: 4,163,353,822 (4.163 GB)
Non-fatal FD errors: 0
SD Errors: 0
FD termination status: OK
SD termination status: OK
Bareos binary info: bareos.org build: Get official binaries and vendor support on bareos.com
Termination: Backup OK
Probably these "extra" files were not sent to FD with accurate data, then FD think they are missing and new, then should be backed up. May be they are sent by DIR, but because of different buffer sizes, just cutted of the accurate data.
As in additional, this happens only in AlwaysInc mode, the classic Full-Diff-Inc works correct.
Andreaas, Jorg, what you think ?
Oleg Volkov
Andreas Rogge
> How can I catch accurate data sent by Director to FD ?
> This can help to understand where is a problem.
director.
With a large enough debug level the client should write the accurate
file list to the dump file.
Bareos GmbH & Co. KG Phone: +49 221-630693-86
http://www.bareos.com
Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Oleg Volkov
--
You received this message because you are subscribed to a topic in the Google Groups "bareos-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bareos-users/PMuqXDqUAWQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bareos-users...@googlegroups.com.
To post to this group, send email to bareos...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/05c5790f-8e29-11fa-eff7-d3be27e7f136%40bareos.com.