Re: [bareos-users] bareos 18.2.5 and 19.2.4~pre992.7dbb7e7be has trouble scaling to many FileSets

[Andreas Rogge]

Hi,

did you also create an individual job for each of your filesets?
Because this should work and if you get a message for the job it will
show up in "list jobs" and you can "list joblog jobid=ID" to look what
went wrong.

Best Regards,
Andreas

Am 14.11.19 um 20:15 schrieb 'Chad Seys' via bareos-users:
> Hi all,
>   I've divided one of our network filesystems into many FileSets on one
> client.  It has 813 FileSets.  We have other clients as well, mostly one
> FileSet per client for a total of 855 FileSets.
>   Should I expect Bareos to be able to handle this many FileSets?
>
>   The problem is that when the scheduled time for all the FileSets
> arrives, almost all are immediately rescheduled for the next scheduled
> time.  (e.g. when today at 12:30pm arrives, 'status dir' shows that they
> are now scheduled for tomorrow at 12:30pm)
>   The jobs show up in the webui as "failure", but there is no other
> message in the webui.
>   'status client' does not show the job ever existed.  (It does not
> appear in list of Terminated Jobs).
>   'list backups client=' also does not show the job.
>   Running the job manually works (after restarting the server daemons).
>
> In the bareos.log file The below message is seen repeatedly.  (Note
> though that Full backups have been taken and an incremental is performed
> with manually run, so  that part is wrong.)
>
> 14-Nov 12:31 bareos-dir JobId 5096: No prior or suitable Full backup
> found in catalog. Doing FULL backup.
> 14-Nov 12:31 bareos-dir JobId 5097: No prior Full backup Job record found.
> 14-Nov 12:31 bareos-dir JobId 5097: No prior or suitable Full backup
> found in catalog. Doing FULL backup.
> 14-Nov 12:31 bareos-dir JobId 5098: No prior Full backup Job record found.
>
> Previous to adding the huge client backups were occurring as expected.
>
> Anyone have ideas, other than trimming back the FileSets?  (It wouldn't
> make much sense to do this. There is so much data / too slow a network
> filesystem for a Full to complete in a reasonable amount of time.)
>
> Thanks!
> Chad.
>  
>
> --
> You received this message because you are subscribed to the Google
> Groups "bareos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to bareos-users...@googlegroups.com
> <mailto:bareos-users...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/bareos-users/f7e04b1e-1253-4941-b882-3707d6b47d12%40googlegroups.com
> <https://groups.google.com/d/msgid/bareos-users/f7e04b1e-1253-4941-b882-3707d6b47d12%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
Andreas Rogge andrea...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221-630693-86
http://www.bareos.com

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: S. Dühr, M. Außendorf, J. Steffens, Philipp Storz

[Chad W Seys]

Hi Andreas,
Yep, each FileSet has a separate job.

Thanks for the tips. I discovered more clues, but backups are still not
running flawlessly. I think the "scale problem" might actually be
simply that bareos is somehow goofed up and doesn't successfully finish
backups (anymore).

I've pared back to just a few clients and have noticed that most of the
"running" backups don't finish successfully. They get stuck in some
state where Some parts of bareos think a job is finished. Other parts
say it is still running and the job never finishes.

This leads to the queued jobs waiting forever.

Here's one example.:

*list jobs jobstatus=running
[...]
| 10,045 | backup-trog.physics.wisc.edu | trog.physics.wisc.edu
| 2019-11-17 15:37:02 | B | I | 0 | 0 | R |
[...]

*status client=trog.physics.wisc.edu
Connecting to Client trog.physics.wisc.edu at trog.physics.wisc.edu:9102
[...]
trog-fd Version: 18.2.5 (30 January 2019) Linux-4.4.92-6.18-default
[...]
Terminated Jobs:
[...]
10045 Incr 50,882 3.717 G OK 17-Nov-19 15:46
backup-trog.physics.wisc.edu

(notice the client says that the backup finished successfully.)
*status storage=PhysHDFS
[...]
Terminated Jobs:
[...]
10045 Incr 50,882 3.740 G OK 17-Nov-19 15:47
backup-trog.physics.wisc.edu
[...]

*list joblog jobid=10045
2019-11-17 15:37:00 bareos-dir JobId 10045: Start Backup JobId 10045,
Job=backup-trog.physics.wisc.edu.2019-11-17_15.37.00_09
2019-11-17 15:37:00 bareos-dir JobId 10045: Connected Storage daemon
at thedoctor.physics.wisc.edu:9103, encryption: TLS_CHACHA20_POLY1305_SHA256
2019-11-17 15:37:00 bareos-dir JobId 10045: Using Device "PhysGdrive1"
to write.
2019-11-17 15:37:00 bareos-dir JobId 10045: Probing client protocol...
(result will be saved until config reload)
2019-11-17 15:37:00 bareos-dir JobId 10045: Connected Client:
trog.physics.wisc.edu at trog.physics.wisc.edu:9102, encryption:
TLS_CHACHA20_POLY1305_SHA256
2019-11-17 15:37:00 bareos-dir JobId 10045: Handshake: Immediate
TLS 2019-11-17 15:37:00 bareos-dir JobId 10045: Encryption:
TLS_CHACHA20_POLY1305_SHA256
2019-11-17 15:37:03 bareos-dir JobId 10045: Sending Accurate information.
2019-11-17 15:37:02 trog-fd JobId 10045: Connected Storage daemon at
thedoctor.physics.wisc.edu:9103, encryption: TLS_CHACHA20_POLY1305_SHA256
2019-11-17 15:38:42 trog-fd JobId 10045: Extended attribute support is
enabled
2019-11-17 15:38:42 trog-fd JobId 10045: ACL support is enabled
2019-11-17 15:38:42 bareos-sd JobId 10045: Spooling data ...
2019-11-17 15:46:22 bareos-sd JobId 10045: Committing spooled data to
Volume "Merged-1547". Despooling 3,747,231,638 bytes ...
2019-11-17 15:47:15 bareos-sd JobId 10045: Despooling elapsed time =
00:00:53, Transfer rate = 70.70 M Bytes/second
2019-11-17 15:47:15 bareos-sd JobId 10045: Releasing device
"PhysGdrive1" (/scratch/gdrive-merged).
2019-11-17 15:47:15 bareos-sd JobId 10045: Elapsed time=00:08:33,
Transfer rate=7.291 M Bytes/second
2019-11-17 15:47:15 bareos-sd JobId 10045: Sending spooled attrs to
the Director. Despooling 15,057,696 bytes ...
2019-11-17 15:47:22 bareos-dir JobId 10045: Insert of attributes batch
table with 2 entries start

Thanks for having a look!
Chad.

[Chad W Seys]

Hi Andreas,
I "fixed" the problem by restoring a dump of the database to a day
before the trouble began. Backups appear to be finishing normally now.
Apparently some junk got into the database and broke bareos.
I haven't yet added the client with hundreds of filesets again.
Waiting for the current backups to finish. I'm hoping that first my
error filled config files polluted bareos db, but now that the bugs are
worked out (?) the same won't happen again.

Thanks!
Chad.