Enhanced configuration scheme in upcoming Bareos version 16.2

[Jörg Steffens]

Hi Bareos users,

as you may have noticed, we are busy to prepare the release of the next
major Bareos version 16.2.

If you already knew Bareos, one thing you will notice immediately after
installing is that the configuration scheme has changed
(https://github.com/bareos/bareos/tree/bareos-16.2 or current master
packages at http://download.bareos.org/bareos/experimental/nightly/)
Prior version had by default one configuration file per Bareos component
(e.g. bareos-dir uses /etc/bareos/bareos-dir.conf, bareos-sd uses
/etc/bareos/bareos-sd.conf, ...).
We noticed that most larger installation splitted there configuration
into several files.
Therefore we decided, to split the Bareos configuration into several
files by default.
This gains us the benefit, that it is easier for Bareos subpackages to
extend the configuration
and also allow us to provide a simple bconsole command to add resources
to the Bareos director configuration,
see
http://doc.bareos.org/master/html/bareos-manual-main-reference.html#sec:bcommandConfigure:

echo "configure add client name=client2−fd address=192.168.0.2
password=secret" | bconsole

The new subdirectory configuration schema uses one file per resource.
So the client client2-fd is stored (by default) in
/etc/bareos/bareos-dir.d/client/client2-fd.conf

As described in
http://doc.bareos.org/master/html/bareos-manual-main-reference.html#SubdirectoryConfigurationScheme,
as long as the old configuration file (e.g. /etc/bareos/bareos-dir.conf)
exists, it will be used.
The only known problem that can arise, if your existing configuration
uses an own wildcard mechanism to include configuration files from
/etc/bareos/bareos-dir.d/ (e.g. @|"sh -c ’echo
/etc/bareos/bareos-dir.d/*.conf'"). In this case, old and new
configuration files may collide. To prevent problems, it is recommended
to backup the /etc/bareos/ configuration directory before updating.

As changing the default configuration scheme has quite some impact when
updating systems, we hope to have implement an approach to combines the
best between offering benefits and minimized the impact on existing
installations.

If anybody of you have improvements to contribute, this may be the last
chance before the release of bareos-16.2 (which is planed for September,
at least a beta release).

regards,
Jörg

--
Jörg Steffens joerg.s...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221 630693-91
http://www.bareos.com Fax: +49 221 630693-10

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer:
S. Dühr, M. Außendorf, Jörg Steffens, P. Storz, M. v. Wieringen

[Jörg Steffens]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Stefan,

I hope, you don't mind, that I share this your important hints also
with the mailinglist.

Am 26.08.2016 um 00:45 schrieb Stefan Klatt:
> Hi Jörg,
>
> I work with the actual betas

thank you for using and testing the nightly builds!

> and found one point with the new configuration schema:
>
> If you install Bareos, it creates a set of default configuration
> files. That's really good to begin with bareos.
>
> But if you work with your own file names like "servername-fd.conf"
> and not "bareos-fd.conf" and you delete the file "bareos-fd" it
> will be new generated if you run an update of Bareos. That's really
> bad, every time after an update Bareos doesn't work any more :-( As
> a workaround I truncate the not needed files with an "#" in it.

That absolutely true, and it is even documented (since beginning of
this week), see
http://doc.bareos.org/master/html/bareos-manual-main-reference.html#Subd
irectoryConfigurationScheme,
Resource file conventions, Disable/remove configuration resource files.

While I agree, that this is an unwanted effect, it got the benefit,
that you can check in your running system if a resource config file
come from a bareos package and is it modified (when using rpm packages).
When this feature is wanted (I like it), you get the described effects.

> I think it's better to generate only the standard directories
> without files and a default set of configuration files with a
> little documentation in an extra example directory structure.

The other feature is, that subpackages can bring additional
configuration, like tape backend, webui and others.
When this feature is wanted, your approach unfortunately do not work.

You see, there is a trade-off between the positive and negative effects.

> One other questions... what's the reason for the directory
> "bareos-dir-export"?

When using the new bconsole command "configure add client ...", it
creates the client resource file for the director.
However, it also creates the director resource for the corresponding
bareos-fd.

So

configure add client name=client2−fd address=192.168.0.2 password=secret

creates

/etc/bareos/bareos-dir.d/client/client2-fd.conf (which is
automatically loaded into the director)

and

/etc/bareos/bareos-dir-export/client/client2-fd/bareos-fd.d/director/bar
eos-dir.conf
(which you have to manually transfer to the corresponding system client2
.

regards,
Jörg
- --

Jörg Steffens joerg.s...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221 630693-91
http://www.bareos.com Fax: +49 221 630693-10

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer:
S. Dühr, M. Außendorf, Jörg Steffens, P. Storz, M. v. Wieringen

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBAgAGBQJXwDYeAAoJENE++pz84GWuk6MQAJRMHjjrwVytYE2x2mh1zFyT
YKoirtsIcveatlN59CKshPI46TzjZcCKT+9+8enMjaUitvnSzyRnBDSMKlvM4RNO
dIzaw3cdbq/ZCzOougtUTJnND1Al7ZNDbEAvHDbNcc1O80CrpZG2X4JUnDkABZs8
W0P2Gl3fE0rdy3mxlDBewFbRce9HIINDTxsGujSgv/w2KxkdWV4owCZsNq/lcmmH
fXPj63jliNVlYkZCrZH90bS4z4M6IVfMv1WT0ID4A0evO2y5xxgW1hayCnnkZ2Ol
bVym+KCECH6psZtkMm7xWtHEvWithIdG4XpOlUR2NJ5cX7UyJvRCzS40+xXQJQYZ
+pexcHCFt5W0bs52lX6PnrS9TF096h/nmHXeFqiN8KeffnvHR6vrDiteWvsFEJu2
329TA/BQRWWwTh2MCKZZZRGtoolKRjkLpbqEYFHvMtjxGZdCoXdXJqxt8raqmvW+
xGGWij+J7e9pbFX0tQmZpNV2hQeUzslaqywGipQF36QnLgudM5pF3TC8AsGgnrWf
cD3ZhOOq2Vta4r8YIbe6i4JKhyIPPOm2QCkl2ntBLsulZRbjURu2bUCILLR0GC8H
9xkpzx5+N7lUiAn/VfS4vo21SnicMNpHzS2+iqSOd/ilFdpjX7JLgcxJootkd51P
AzvWDRsvhcWEeWlVz/3X
=x3Y1
-----END PGP SIGNATURE-----

[Stefan Klatt]

Hi Jörg,

> > I work with the actual betas
> thank you for using and testing the nightly builds!

Sometimes they are a little tricky. I didn't got one of the last Windows
Client working.

> > and found one point with the new configuration schema:
> > If you install Bareos, it creates a set of default configuration
> > files. That's really good to begin with bareos.
> > But if you work with your own file names like "servername-fd.conf"
> > and not "bareos-fd.conf" and you delete the file "bareos-fd" it
> > will be new generated if you run an update of Bareos. That's really
> > bad, every time after an update Bareos doesn't work any more :-( As
> > a workaround I truncate the not needed files with an "#" in it.
> That absolutely true, and it is even documented (since beginning of
> this week), see
> http://doc.bareos.org/master/html/bareos-manual-main-reference.html#Subd

> irectoryConfigurationScheme,Resource file conventions, Disable/remove

> configuration resource files.
>
> While I agree, that this is an unwanted effect, it got the benefit,
> that you can check in your running system if a resource config file
> come from a bareos package and is it modified (when using rpm packages).
> When this feature is wanted (I like it), you get the described effects.

I don't think so because automatic implementation of new features or
configuration changes brake often existing configurations.
They should only implemented manually after review.

> > I think it's better to generate only the standard directories
> > without files and a default set of configuration files with a
> > little documentation in an extra example directory structure.
>
> The other feature is, that subpackages can bring additional
> configuration, like tape backend, webui and others.
> When this feature is wanted, your approach unfortunately do not work.

Here the same... nothing is bad like a crashed system and you search a
unwanted change.
They should only implemented manually after review.

> You see, there is a trade-off between the positive and negative effects.

I don't think so because the goal is a running system under each condition.
If I make an update and nothing work any more after this, horrible!

> > One other questions... what's the reason for the directory
> > "bareos-dir-export"?
>
> When using the new bconsole command "configure add client ...", it
> creates the client resource file for the director.
> However, it also creates the director resource for the corresponding
> bareos-fd.
> So
> configure add client name=client2−fd address=192.168.0.2 password=secret
>
> creates
> /etc/bareos/bareos-dir.d/client/client2-fd.conf (which is
> automatically loaded into the director)
>
> and
> /etc/bareos/bareos-dir-export/client/client2-fd/bareos-fd.d/director/bar
> eos-dir.conf
> (which you have to manually transfer to the corresponding system client2

For this I use an ansible script.

Regards

Stefan

--
*CaC, Computer and Communication*
Inhaber Stefan Klatt
End-2-End Senior Network Consultant
Triftstrasse 9
60528 Frankfurt
Germany
USt-IdNr.: DE260461592

Tel.: +49-(0)172-6807809
Tel.: +49-(0)69-67808-900
Fax: +49-(0)69-67808-837
Email: stefan...@cac-netzwerk.de
Profil: http://www.cac-netzwerk.de/profil

[Jörg Steffens]

First of all, I totally agree that an update should never break a system
configuration. So maintenance releases for the different Bareos branches
(12.4, 13.2, 14.2, 15.2 and soon 16.2) are bugfixes or improvements that
do not change the default behavior.

In master, changes can happen, as we also have to learn and test how
things works out best.

I understand your point of view, however I do not see this big impact,
as adding a resource (e.g. a profile) will not change Bareos behavior.
Of course, in principle we can add resources that gets automatically
scheduled, however we are not planing to do so.
And we are trying to make things easier for most people, as over the
years I've heard several times complains that resources did not get
loaded automatically.

As an administrator you still got the chance to disable this behavior,
by using a non-default Bareos configuration directory (either change the
service file or create a /etc/bareos/bareos-dir.conf with following
content: @/etc/my-bareos-config/bareos-dir.d/*/*.conf).

Anyhow, of course we like to implement what all (or at least) most
people want. As said, in the past, I heard several persons wanting the
feature as implemented right now.
Anybody else with an opinion on this?

regards,
Jörg

[Bruno Friedmann]

Playing since some weeks with the 16.2 future configuration layout (Preparing the french training) even with a 15.2 is a pleasure to use.
So I transfert my lnog old way of doing things inside the new way, and everything is working well.
Now if 16.2 rpm (those I'm using) will broke MY configuration in anyway I would be really disapointed, after years of smooth upgrade.

I guess I didn't have yet understood how it will impact things. and I have to certainly spend more times on 16.2 (but time is flying as always :-))

I'm expecting that on 16.2 it will just use what I give to it, the new structure (migrated by hand once) and just play nicely with.

I deeply hope that the way *.conf are handled in packaging are still the same, like if it exist, don't touch and create a rpm(deb).new config file.

Of course I can move my private things to another directory, but I don't think it is the best idea to have.

The proposed structure is really nice, and with the coming api configuration ability it a great step forwards.

I also undestand that I will have to at least let now bareos-dir.d directory writeable by bareos users. I tend to be more strict in the configuration directory that only root or dedicated owner can write and bareos user can only read.

Anyway I'm looking forward this new version, and fantastic time at osbconf in a few weeks.

[Jörg Steffens]

Am 29.08.2016 um 22:28 schrieb Bruno Friedmann:
[...]

> I'm expecting that on 16.2 it will just use what I give to it, the
> new structure (migrated by hand once) and just play nicely with.

yes. I'll work with old bareos-*.conf config files and the bareos-*.d
configuration directory.

> I deeply hope that the way *.conf are handled in packaging are still
> the same, like if it exist, don't touch and create a rpm(deb).new
> config file.

Yes, on RPM all config files are tagged with %config(noreplace). This
should avoid that your changes get overwritten.
On Debian things work differently (packages configs are stored in
/usr/lib/bareos/defaultconfigs/. In postinstall, files not existing in
/etc/bareos/ are copied over from /usr/lib/bareos/defaultconfigs/). On
Windows, things are done similar to Debian, besides that you can
configure settings during installation and that the deploy script is
written in DOS Batch, with all its limitations.

Extra packages can bring extra configuration resources, either as *.conf
or as *.conf.example files. In theory these can create problems. The
Bareos project is responsible to only add harmless resources.

> Of course I can move my private things to another directory, but I
> don't think it is the best idea to have.
>
> The proposed structure is really nice, and with the coming api
> configuration ability it a great step forwards.
>
> I also undestand that I will have to at least let now bareos-dir.d
> directory writeable by bareos users.

That is true.

> I tend to be more strict in the
> configuration directory that only root or dedicated owner can write
> and bareos user can only read.

Understood. However, I'm not aware of a better approach to solve this.

> Anyway I'm looking forward this new version, and fantastic time at
> osbconf in a few weeks.

Thanks! I'm also looking forward.

[Stefan Klatt]

Hi Jörg,

> As an administrator you still got the chance to disable this behavior,
> by using a non-default Bareos configuration directory (either change the
> service file or create a /etc/bareos/bareos-dir.conf with following
> content: @/etc/my-bareos-config/bareos-dir.d/*/*.conf).

A good idea... but I think this will break the new feature "configure
add client" you described.

> Anyhow, of course we like to implement what all (or at least) most
> people want. As said, in the past, I heard several persons wanting the
> feature as implemented right now.

I believe this. With the single config file there was no problem, but
with the new config layout....

[Stefan Klatt]

Hi Bruno,

> I also undestand that I will have to at least let now bareos-dir.d directory writeable by bareos users. I tend to be more strict in the configuration directory that only root or dedicated owner can write and bareos user can only read.

Like I understand bareos need minimum write rights to
"bareos-dir.d/client" if you use the new bconsole function "configure
add client".

> Anyway I'm looking forward this new version, and fantastic time at osbconf in a few weeks.

The new version is a really good big step in the right direction.

[Stefan Klatt]

Hi Jörg,

> I understand your point of view, however I do not see this big impact,
> as adding a resource (e.g. a profile) will not change Bareos behavior.

I use the config files "Daemon.conf" and "Standard.conf" for message
configuration. If I update bareos it writes it's own message
configuration and brakes the running configuration.
This is only an example, I had a views problems to correct my
configuration after last update.

> Of course, in principle we can add resources that gets automatically
> scheduled, however we are not planing to do so.
> And we are trying to make things easier for most people, as over the
> years I've heard several times complains that resources did not get
> loaded automatically.

This works only in small systems but not with a bigger configuration.

[Jörg Steffens]

Am 29.08.2016 um 23:41 schrieb Stefan Klatt:
>> I understand your point of view, however I do not see this big impact,
>> as adding a resource (e.g. a profile) will not change Bareos behavior.
> I use the config files "Daemon.conf" and "Standard.conf" for message
> configuration. If I update bareos it writes it's own message
> configuration and brakes the running configuration.

Let me try to understand this:
You manually created configuration files
/etc/bareos/bareos-dir.d/messages/Daemon.conf on a Bareos installation,
which did not include this file and than updated to a Bareos version
which includes this file and the Bareos package has overwritten your
file? Correct?
If yes, than this would be indeed be a problem. On what OS are you
running Bareos?

[Jörg Steffens]

Am 29.08.2016 um 23:06 schrieb Stefan Klatt:
> Hi Jörg,
>
>> As an administrator you still got the chance to disable this behavior,
>> by using a non-default Bareos configuration directory (either change the
>> service file or create a /etc/bareos/bareos-dir.conf with following
>> content: @/etc/my-bareos-config/bareos-dir.d/*/*.conf).
> A good idea... but I think this will break the new feature "configure
> add client" you described.

True. However, I'll work if your start bareos-dir with the option "-c
/etc/my-bareos-config/".

[Stefan Klatt]

Hi Jörg,

Am 31.08.2016 um 15:38 schrieb Jörg Steffens:

> Am 29.08.2016 um 23:06 schrieb Stefan Klatt:
>> Hi Jörg,
>>
>>> As an administrator you still got the chance to disable this behavior,
>>> by using a non-default Bareos configuration directory (either change the
>>> service file or create a /etc/bareos/bareos-dir.conf with following
>>> content: @/etc/my-bareos-config/bareos-dir.d/*/*.conf).
>> A good idea... but I think this will break the new feature "configure
>> add client" you described.
> True. However, I'll work if your start bareos-dir with the option "-c
> /etc/my-bareos-config/".

That's bad because you can't work with bareos-dir without wrapper.

What about a optional configurationfile which includes only the pathes
to the configuration files?

[Stefan Klatt]

Hi Jörg,

let me describe this on another way:

I use my own set of configuration files that are more divided, e.g.:

bareos org config -> config file 1, config file 2

bareos updates creates the original file. And now bareos can't start if
the ressource can't be double like messages....

Other effect is I have a client "bareos-fd" in my database that doesn't
exist. The same to pools and schedules.

Regards

Stefan

Am 31.08.2016 um 15:33 schrieb Jörg Steffens:

> Am 29.08.2016 um 23:41 schrieb Stefan Klatt:
>>> I understand your point of view, however I do not see this big impact,
>>> as adding a resource (e.g. a profile) will not change Bareos behavior.
>> I use the config files "Daemon.conf" and "Standard.conf" for message
>> configuration. If I update bareos it writes it's own message
>> configuration and brakes the running configuration.
> Let me try to understand this:
> You manually created configuration files
> /etc/bareos/bareos-dir.d/messages/Daemon.conf on a Bareos installation,
> which did not include this file and than updated to a Bareos version
> which includes this file and the Bareos package has overwritten your
> file? Correct?
> If yes, than this would be indeed be a problem. On what OS are you
> running Bareos?
>
> regards,
> Jörg
>
>

[Kjetil Torgrim Homme]

Jörg Steffens <joerg.s...@bareos.com> writes:

> As an administrator you still got the chance to disable this behavior,
> by using a non-default Bareos configuration directory (either change the
> service file or create a /etc/bareos/bareos-dir.conf with following
> content: @/etc/my-bareos-config/bareos-dir.d/*/*.conf).
>
> Anyhow, of course we like to implement what all (or at least) most
> people want. As said, in the past, I heard several persons wanting the
> feature as implemented right now.
> Anybody else with an opinion on this?

I like the new scheme, although I will not use it myself since I manage
clients and jobs via Puppet. however, it is annoying that when I do
"make install" from the git master, I get a lot of gunk in /etc/bareos.

it would be nice if configure had a switch to disable the installation
of default configuration, or even better, if it installed it to
"$docdir/example-config/" and left it completely up to packagers what to
put in "/etc/bareos".

FWIW, I currently use these path related settings with configure:

./configure --prefix=/opt/bareos --with-confdir=/etc/bareos --with-scriptdir=/etc/bareos/distscripts

(the latter setting helps a lot, and I certainly don't want my finely
tuned query.sql to be garbled after each make install ;)

--
Kjetil T. Homme
Redpill Linpro AS - Changing the game

[Jörg Steffens]

Am 07.09.2016 um 20:17 schrieb Kjetil Torgrim Homme:
> I like the new scheme, although I will not use it myself since I manage
> clients and jobs via Puppet. however, it is annoying that when I do
> "make install" from the git master, I get a lot of gunk in /etc/bareos.
>
> it would be nice if configure had a switch to disable the installation
> of default configuration, or even better, if it installed it to
> "$docdir/example-config/" and left it completely up to packagers what to
> put in "/etc/bareos".
>
> FWIW, I currently use these path related settings with configure:
>
> ./configure --prefix=/opt/bareos --with-confdir=/etc/bareos --with-scriptdir=/etc/bareos/distscripts
>
> (the latter setting helps a lot, and I certainly don't want my finely
> tuned query.sql to be garbled after each make install ;)

So, configure already offers the functionality you suggested. Or am I
missing something? (Patches welcome)

[Kjetil Torgrim Homme]

Jörg Steffens <joerg.s...@bareos.com> writes:

> Am 07.09.2016 um 20:17 schrieb Kjetil Torgrim Homme:
>> I like the new scheme, although I will not use it myself since I manage
>> clients and jobs via Puppet. however, it is annoying that when I do
>> "make install" from the git master, I get a lot of gunk in /etc/bareos.
>>
>> it would be nice if configure had a switch to disable the installation
>> of default configuration, or even better, if it installed it to
>> "$docdir/example-config/" and left it completely up to packagers what to
>> put in "/etc/bareos".
>>
>> FWIW, I currently use these path related settings with configure:
>>
>> ./configure --prefix=/opt/bareos --with-confdir=/etc/bareos --with-scriptdir=/etc/bareos/distscripts
>>
>> (the latter setting helps a lot, and I certainly don't want my finely
>> tuned query.sql to be garbled after each make install ;)
>
> So, configure already offers the functionality you suggested. Or am I
> missing something? (Patches welcome)

yes and no, I still get the bareos-XX.d directories. but thanks for the
prod, I looked a little closer and found that debian.rules uses:

--with-configtemplatedir=/usr/lib/bareos/defaultconfigs

so that would solve my problem, except it is only used in the files in
debian/. the default is a bit strange, too:

autoconf/configure.in:configtemplatedir=`eval echo ${libdir}`

so this should change to confdir (Debian overrides the default anyhow),
and configtemplatedir should be added to autoconf/Make.common.in

patch:

[Jörg Steffens]

Am 08.09.2016 um 15:02 schrieb Kjetil Torgrim Homme:
[...]

> yes and no, I still get the bareos-XX.d directories. but thanks for the
> prod, I looked a little closer and found that debian.rules uses:
>
> --with-configtemplatedir=/usr/lib/bareos/defaultconfigs
>
> so that would solve my problem, except it is only used in the files in
> debian/. the default is a bit strange, too:
>
> autoconf/configure.in:configtemplatedir=`eval echo ${libdir}`
>
> so this should change to confdir (Debian overrides the default anyhow),
> and configtemplatedir should be added to autoconf/Make.common.in

Thank you for the patch and sorry for the long delay. I've been busy
with osbconf.org and the preparation of the bareos-16.2 release.

I understand that your patch can be useful if people are compiling
Bareos manually.

However, the patch breaks building Debian packages. While trying to fix
this, I noticed, that the patch is incomplete. It seams that at least
handling of plugin config files is not covered (src/plugins/*). Also the
client programs (bconsole, bat, tray-monitor) are not covered.

I'm open for accepting a fixed version of this patch.
I assume, the adaption in debian/* can be done by:

sed -i -r 's/@confdir@([^ ]*).*/@configtemplatedir@\1/' debian/*.install.in

regards,