Restore job - selection of files takes a VERY long time

[Marco Hoppstädter]

Hello,

I've asked this question before but got no response. I am trying to
restore some files from a large backupjob (~ 600GB) with a lot of files
(~ 4.7 million). When selecting the backup job in the WebUI the
selection of the files seems to take forever (at least two hours or so).
So is this normal behaviour? Or can anyone give me a clue how to speed
this up?

Thanks in advance for any help!

Freundliche Grüße aus Saarbrücken

----------------------------------------------------------------------
Marco Hoppstädter Email: marco.hop...@hightec-rt.com
HighTec EDV-Systeme GmbH Tel: +49-681-92613-38
Feldmannstrasse 98 WWW: http://www.hightec-rt.com
D-66119 Saarbrücken
----------------------------------------------------------------------
Managing Directors: Vera Strothmann, Dr. Rolf Strothmann
Local Court: Saarbrücken, HRB 10445, VAT ID: DE 138344222

This e-mail may contain confidential and/or privileged information. If
you are not the intended recipient please notify the sender immediately
and destroy this e-mail. Any unauthorised copying, disclosure or
distribution of the material in this e-mail is strictly forbidden.
---

--
---This email was virus checked by HighTec mail gateway---

[Marco Hoppstädter]

Some additional info:

Bareos is running in version 15.2.2-37.1 on Ubuntu 16.04.1. Server is a
quad core Xeon X3450 @ 2.67GHz, 14 GB RAM. System and database on
Hardware RAID1.
When bareos file selection is running

- one core is at 100% (postgres: bareos bareos [local] INSERT)
- RAM is not fully used, no swap
- disk i/o is low

See screenshot, if screenshots are possible.


Best regards.

[Bruno Friedmann]

On mardi, 23 août 2016 13.15:03 h CEST Marco Hoppstädter wrote:
> Some additional info:
>
> Bareos is running in version 15.2.2-37.1 on Ubuntu 16.04.1. Server is a
> quad core Xeon X3450 @ 2.67GHz, 14 GB RAM. System and database on
> Hardware RAID1.
> When bareos file selection is running
>
> - one core is at 100% (postgres: bareos bareos [local] INSERT)
> - RAM is not fully used, no swap
> - disk i/o is low
>
> See screenshot, if screenshots are possible.
>
>
> Best regards.
>
> Am 23.08.2016 um 12:22 schrieb Marco Hoppstädter:
> > Hello,
> >
> > I've asked this question before but got no response. I am trying to
> > restore some files from a large backupjob (~ 600GB) with a lot of files
> > (~ 4.7 million). When selecting the backup job in the WebUI the
> > selection of the files seems to take forever (at least two hours or so).
> > So is this normal behaviour? Or can anyone give me a clue how to speed
> > this up?
> >
> > Thanks in advance for any help!
> >

Hi Marco,

~300 MB for postgresql ? I hope you're just joking with us :-)

To be short allow 1/4 of ram to postgresql to get it running better
Use help from http://pgtune.leopard.in.ua/

a quick check give these configuration


max_connections = 32
shared_buffers = 3584MB
effective_cache_size = 10752MB
work_mem = 224MB
maintenance_work_mem = 1792MB
checkpoint_segments = 128
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 500

etc, there's plenty of documentation or professional consulting on the web
to help you to setup postgresql correctly.

;-)

--

Bruno Friedmann
Ioda-Net Sàrl www.ioda-net.ch
Bareos Partner, openSUSE Member, fsfe fellowship
GPG KEY : D5C9B751C4653227
irc: tigerfoot

openSUSE Tumbleweed (20160820) (x86_64)
Linux 4.6.4-2-default x86_64 GNU/Linux, nvidia: 370.23
Qt: 5.6.1, KDE Frameworks: 5.24.0, Plasma: 5.7.3, kmail2 5.2.3

[Marco Hoppstädter]

> Hi Marco,
>
> ~300 MB for postgresql ? I hope you're just joking with us :-)
>
> To be short allow 1/4 of ram to postgresql to get it running better
> Use help from http://pgtune.leopard.in.ua/
>
> a quick check give these configuration
>
>
> max_connections = 32
> shared_buffers = 3584MB
> effective_cache_size = 10752MB
> work_mem = 224MB
> maintenance_work_mem = 1792MB
> checkpoint_segments = 128
> checkpoint_completion_target = 0.9
> wal_buffers = 16MB
> default_statistics_target = 500
>
> etc, there's plenty of documentation or professional consulting on the web
> to help you to setup postgresql correctly.
>
> ;-)
>

Hi Bruno,

thanks for the helpful link. I've searched (and found) some information
about tuning PostgreSQL yesterday and already tried some modifications
on the config, but not that successful. Now I will try your suggestion.
Thanks.

Best regards
Marco

[Marco Hoppstädter]

Hi,

now I've played around on http://pgtune.leopard.in.ua/ and configured
these settings:

max_connections = 32
shared_buffers = 3584MB
effective_cache_size = 10752MB
work_mem = 224MB
maintenance_work_mem = 1792MB

checkpoint_completion_target = 0.9
maintenance_work_mem = 896MB
min_wal_size = 1GB
max_wal_size = 2GB
wal_buffers = 16MB
default_statistics_target = 100


I started the file selection in the restore section of Bareos bat and
it's running now for about half an hour. In top postgres is not using
that much memory as before (see screenshot). So one question I have is
about the max connections. Only bareos uses the postgres database, so
would it be better to set max connections to a low value and therefore a
higher work_mem value or the other way around?

Thanks.
Marco

[Hubert Gilch]

Hello Marco,

i found that selection in the web ui always takes a very long time, even with smaller jobs, wheras the same selection in bconsole takes just some 10 seconds.

Since i start restores via bconsole, every restore is just a matter of minutes (with only a few files, of course).

Some additional info:

Bareos is running in version 15.2.2-37.1 on Ubuntu 16.04.1. Server is a
quad core Xeon X3450 @ 2.67GHz, 14 GB RAM. System and database on
Hardware RAID1.
When bareos file selection is running

- one core is at 100% (postgres: bareos bareos [local] INSERT)
- RAM is not fully used, no swap
- disk i/o is low

See screenshot, if screenshots are possible.


Best regards.


Am 23.08.2016 um 12:22 schrieb Marco Hoppstädter:

Hello,

I've asked this question before but got no response. I am trying to
restore some files from a large backupjob (~ 600GB) with a lot of files
(~ 4.7 million). When selecting the backup job in the WebUI the
selection of the files seems to take forever (at least two hours or so).
So is this normal behaviour? Or can anyone give me a clue how to speed
this up?

Thanks in advance for any help!

Mit freundlichem Gruß
i. A. Hubert Gilch

Systemadministrator

SEP Logistik AG
Ziegelstr. 2
D-83629 Weyarn
Tel.:     +49 8020 905-214
Fax:     +49 8020 905-100
Email: h...@sepag.de
Web: www.sepag.de


Sitz der Gesellschaft: Weyarn Rechtsform: Aktiengesellschaft Amtsgericht München: HRB 119679
Vorstand: Professor Dr.-Ing. Franz Demmelmeier (Vorsitzender), Dipl.-Ing. Konrad Schneider, MBA, Dipl.-Ing. Markus Fischbacher
Aufsichtsratsvorsitzender Michael Pelzer

[Jörg Steffens]

Am 24.08.2016 um 09:28 schrieb Hubert Gilch:
> Hello Marco,
>
> i found that selection in the web ui always takes a very long time, even
> with smaller jobs, wheras the same selection in bconsole takes just some
> 10 seconds.

bconsole uses a different set of functions to get the file information.
bat and bareos-webui use the .bvfs_* commands.

Yes, we are aware that the database performance can get a bottleneck, if
you have a very large filename table (a lot of different files in all
your current backup).

We are currently engaged in a co-founding project with a customer for
improving Bareos database performance. We tried different approaches and
have now a prototype, which largely (!!!) improves performance, see
https://www.bareos.com/en/co-funding.html#performance-enhancements

So you either have to wait that this gets into the official release (it
will not be ready for the the upcoming 16.2 release) or may engage in
the co-founding process yourself.

regards,
Jörg


--
Jörg Steffens joerg.s...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221 630693-91
http://www.bareos.com Fax: +49 221 630693-10

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer:
S. Dühr, M. Außendorf, Jörg Steffens, P. Storz, M. v. Wieringen