Re: declarative (config file) way idea of handling the OS by way of the old system

[Emanuel Berg]

> Anyone has that for Debian and Debian-like systems?

Or, to be exact, systems that use APT.

--
underground experts united
https://dataswamp.org/~incal

[Emanuel Berg]

Have a look at this blog post and program:

Managing OpenBSD installed packages declaratively
https://dataswamp.org/~solene/2022-05-05-openbsd-declarative-packages-with-pkgset.html

That's sure one idea, to have a config file (that's the
"declarative" part) and then a script that converts that into
commands of/to the traditional package manager to setup the
system in accordance ...

Anyone has that for Debian and Debian-like systems?

[Mike Kupfer]

Emanuel Berg wrote:

> Have a look at this blog post and program:
>
> Managing OpenBSD installed packages declaratively
> https://dataswamp.org/~solene/2022-05-05-openbsd-declarative-packages-with-pkgset.html

[...]

> Anyone has that for Debian and Debian-like systems?

I have half of that, sort of. A static file has 1 package name per
line, and each line has a usage tag like "base", "dev", or
"emacs-build", and distro tags, like "deb10" or "f25". I run a script,
telling it the usage and distro, and it spits out a list of packages. I
usually redirect the list to a file, review and maybe edit it, and do
something like "apt install $(cat tmp/pkgs)".

I don't have the half that deletes unwanted packages, but that's not
functionality I feel a need for.

regards,
mike

[Emanuel Berg]

Indeed, sounds like you have the essential half ...

[Timothy M Butterworth]

On Sun, May 29, 2022 at 11:10 PM Emanuel Berg <in...@dataswamp.org> wrote:

Have a look at this blog post and program:

  Managing OpenBSD installed packages declaratively
  https://dataswamp.org/~solene/2022-05-05-openbsd-declarative-packages-with-pkgset.html

That's sure one idea, to have a config file (that's the
"declarative" part) and then a script that converts that into
commands of/to the traditional package manager to setup the
system in accordance ...

Isn't that essentially how puppet works? I know it has a config file that translates into commands and it can be used to install software packages.

[Dan Ritter]

Emanuel Berg wrote:
> Have a look at this blog post and program:
>
> Managing OpenBSD installed packages declaratively
> https://dataswamp.org/~solene/2022-05-05-openbsd-declarative-packages-with-pkgset.html
>
> That's sure one idea, to have a config file (that's the
> "declarative" part) and then a script that converts that into
> commands of/to the traditional package manager to setup the
> system in accordance ...
>
> Anyone has that for Debian and Debian-like systems?

You can do it with puppet, chef, ansible, salt...

You can go further into it with guix and nix.

It can be quite a lot of work, which is generally worthwhile if
you have many systems that need to be centrally controlled and
are similar to each other, or if you need to accurately
replicate a system many times.

For a single user's machine, it's unlikely to be rewarding
except intellectually.

-dsr-

[IL Ka]

You are probably looking for unattended installation and configuration using "noninteractive" dbconf frontend.

https://manpages.debian.org/bullseye/debconf-doc/debconf.7.en.html#Unattended_Package_Installation

[Emanuel Berg]

Dan Ritter wrote:

> You can do it with puppet, chef, ansible, salt...
>
> You can go further into it with guix and nix.
>
> It can be quite a lot of work

Okay, but why so, since it looks like the task to be done is
just a config file that's iterated by a script and boiled down
to suitable commands that are well-known, not the least
because we are used to do them manually?

Yes, ironic ...

[Emanuel Berg]

Mike Kupfer wrote:

> I have half of that, sort of. A static file has 1 package
> name per line, and each line has a usage tag like "base",
> "dev", or "emacs-build", and distro tags, like "deb10" or
> "f25". I run a script, telling it the usage and distro, and
> it spits out a list of packages. I usually redirect the list
> to a file, review and maybe edit it, and do something like
> "apt install $(cat tmp/pkgs)".
>
> I don't have the half that deletes unwanted packages, but
> that's not functionality I feel a need for.

Feel free to share, I forgot to say.

[Emanuel Berg]

IL Ka wrote:

> You are probably looking for unattended installation and
> configuration using "noninteractive" dbconf frontend.
> https://manpages.debian.org/bullseye/debconf-doc/debconf.7.en.html#Unattended_Package_Installation

Noninteractive is the word! Thank you.

But obviously people can call it declarative if the like to,
maybe a fancy word like that is what it takes to promote it.
Now why every that would be needed, since it's such a basic
idea (the config file) and it's present for everything else
already ... but evidently.

Anyway how does that work in practice? Maybe it is even
described in the man page you refer to ...

[Dan Ritter]

Emanuel Berg wrote:
> Dan Ritter wrote:
>
> > You can do it with puppet, chef, ansible, salt...
> >
> > You can go further into it with guix and nix.
> >
> > It can be quite a lot of work
>
> Okay, but why so, since it looks like the task to be done is
> just a config file that's iterated by a script and boiled down
> to suitable commands that are well-known, not the least
> because we are used to do them manually?

For the very obvious reason that there is one major additional
task to go from "I have configured this machine the way I want
it" to "This machine's configuration is now encoded in a
language I needed to learn for the purpose" -- and a second task
to go to "and it can be generalized to run on multiple
machines".

So if you're doing it for one machine, you effectively have to
do it twice; if you're doing it for N machines, you have to do
it about 2 and a half times. If N is much larger than 3, you
win.

-dsr-

[Emanuel Berg]

... what do you mean?

It's the same package manager on all N system, you only need
to do it once.

Also, there is no need to encode the configuration in any
advanced "language", it can be basic configuration like for
xpdf or mpv or whatever.

OTOH if you want a real programming language and environment
for configuration and extension (e.g., Elisp for Emacs, Lua
for mpv etc) that's doable as well, no doubt.

[IL Ka]

Noninteractive is the word! Thank you.

This is a technical term covered by debconf(7) :)

https://manpages.debian.org/bullseye/debconf-doc/debconf.7.en.html

The idea is covered here:

https://manpages.debian.org/bullseye/debconf-doc/debconf-devel.7.en.html

With a non-interactive frontend you can install all packages and configure them with one script.

If you want to patch configuration (files in ``/etc``) you can use either patch(1) or some VCS (like git).

 

But obviously people can call it declarative if the like to,
maybe a fancy word like that is what it takes to promote it.

There is a term "Configuration as code" (or "infrastructure as code"). The idea is to describe your configuration using a text file, store it in VCS and apply.

Lots of tools are available:

* Ansible

* Chef

* Puppet

* Terraform

 

Anyway how does that work in practice? Maybe it is even
described in the man page you refer to ...

debconf is covered in manpage. For best practices google for "configuration as code".

There is a good explanation, and even a book!

 

https://martinfowler.com/bliki/InfrastructureAsCode.html 

[Emanuel Berg]

IL Ka wrote:

> With a non-interactive frontend you can install all packages
> and configure them with one script.

Right, and I happen to know exactly what I want, I want - from
the repos, with the default configuration - if by
configuration you mean install options - these programs

feh
mpv
openbsd-cwm
picom
rtorrent
tmux
xpdf
xterm
zsh

I also want Emacs but I'll install that myself thank you since
I like a more recent version than the one found in the repos.
But I used that or the corresponding ones during that period
for several years so I know there's nothing wrong with it.

Any idea how that would look? I mean the feh-zsh list.

> There is a term "Configuration as code"

To me it's just a configuration file like all software has or
should have (like a man page and --help and --version
options), that's the first step, if you then would like to go
on and have a real programming language to do it (and more,
better etc) like Elisp for Emacs, Lua for mpv etc that sounds
great to me! CL for Debian maybe? Or Python since everyone
uses that. Or both, only not in combination ...

> There is a good explanation, and even a book!
> https://martinfowler.com/bliki/InfrastructureAsCode.html

Amazing!

[Dan Ritter]

Emanuel Berg wrote:
> Dan Ritter wrote:
>
> >>> You can do it with puppet, chef, ansible, salt...
> >>>
> >>> You can go further into it with guix and nix.
> >>>
> >>> It can be quite a lot of work
> >>
> >> Okay, but why so, since it looks like the task to be done
> >> is just a config file that's iterated by a script and
> >> boiled down to suitable commands that are well-known, not
> >> the least because we are used to do them manually?
> >
> > For the very obvious reason that there is one major
> > additional task to go from "I have configured this machine
> > the way I want it" to "This machine's configuration is now
> > encoded in a language I needed to learn for the purpose" --
> > and a second task to go to "and it can be generalized to run
> > on multiple machines".
> >
> > So if you're doing it for one machine, you effectively have
> > to do it twice; if you're doing it for N machines, you have
> > to do it about 2 and a half times. If N is much larger than
> > 3, you win.
>
> ... what do you mean?
>
> It's the same package manager on all N system, you only need
> to do it once.

That's just knowing what packages you want to install. If that's
all you want, you can use dpkg --set-selections and a text list.

> Also, there is no need to encode the configuration in any
> advanced "language", it can be basic configuration like for
> xpdf or mpv or whatever.
>
> OTOH if you want a real programming language and environment
> for configuration and extension (e.g., Elisp for Emacs, Lua
> for mpv etc) that's doable as well, no doubt.

chef, puppet, ansible and so forth are languages for doing that.


-dsr-

[Emanuel Berg]

Dan Ritter wrote:

> That's just knowing what packages you want to install.

What do you mean, what else are you supposed to know?

> If that's all you want, you can use dpkg --set-selections
> and a text list.

What about using the interactive commands?

sudo apt-get -qq update
sudo apt-get install build-essential debian-goodies libgccjit-10-dev

?

>> OTOH if you want a real programming language and
>> environment for configuration and extension (e.g., Elisp
>> for Emacs, Lua for mpv etc) that's doable as well,
>> no doubt.
>
> chef, puppet, ansible and so forth are languages for
> doing that.

You want a language for doing that? Why not a general-purpose
language either adapted if necessary (a dialect) or better yet
extended with a module to do it so you can use the rest of the
language as well seamlessly?

Or what does chef, puppet and ansible have, a special integer
data type to denote semantic versioning and a built-in
implementation of the popular "Old English" algorithm to
resolve circular dependencies?

[Dan Ritter]

Emanuel Berg wrote:
> Dan Ritter wrote:
>
> > That's just knowing what packages you want to install.
>
> What do you mean, what else are you supposed to know?

Examples:

- you have installed a load balancer; it needs a configuration
file to work.

- you have installed bind or unbound to provide DNS, they need
configuration files to work.

- you have installed a database, it may need tuning.

- you have an NFS server on your network, each machine needs a
new stanza in /etc/fstab to mount it except the NFS server

- you have a webserver, it needs configuration and a set of
files to serve

> > If that's all you want, you can use dpkg --set-selections
> > and a text list.
>
> What about using the interactive commands?
>
> sudo apt-get -qq update
> sudo apt-get install build-essential debian-goodies libgccjit-10-dev

Is there a reason that you interpret everything I say as a
challenge?

Sure, you can also do package installation that way. Package
installation is not the end of most matters, it's the beginning.

> >> OTOH if you want a real programming language and
> >> environment for configuration and extension (e.g., Elisp
> >> for Emacs, Lua for mpv etc) that's doable as well,
> >> no doubt.
> >
> > chef, puppet, ansible and so forth are languages for
> > doing that.
>
> You want a language for doing that? Why not a general-purpose
> language either adapted if necessary (a dialect) or better yet
> extended with a module to do it so you can use the rest of the
> language as well seamlessly?

Chef uses ruby with adaptations.

Puppet uses a DSL written in ruby.

ansible is entirely declarative, in YAML

-dsr-

[Emanuel Berg]

Dan Ritter wrote:

>>> That's just knowing what packages you want to install.
>>
>> What do you mean, what else are you supposed to know?
>
> Examples:
>
> - you have installed a load balancer; it needs
> a configuration file to work.
>
> - you have installed bind or unbound to provide DNS, they
> need configuration files to work.
>
> - you have installed a database, it may need tuning.
>
> - you have an NFS server on your network, each machine needs
> a new stanza in /etc/fstab to mount it except the NFS
> server
>
> - you have a webserver, it needs configuration and a set of
> files to serve

Okay, right, no here we're only concerned with the state of
the OS in terms of packages that are installed so that they
can be used immediately by the user, I know that all software
can be and is by some including me configured with no end in
sight but I see no reason to bring _that_ into _this_, since
that is already done with config files and that's the best way
there is to do it IMO, and now it's here as well - well, not
really, that's what I'm asking for - but when it is for me as
well, that'd be then end of it for me and I see no reason to
mix it all together, plus there are other ways to automate
bringing a bunch of files together on a disk if it comes
to that.

>> > If that's all you want, you can use dpkg --set-selections
>> > and a text list.
>>
>> What about using the interactive commands?
>>
>> sudo apt-get -qq update
>> sudo apt-get install build-essential debian-goodies libgccjit-10-dev
>
> Is there a reason that you interpret everything I say as a
> challenge?

If you just want certain packages installed, you can
have for example a shell function in your shell language, e.g.
zsh with an array of packs and then a command, e.g.

#! /bin/zsh

install-os () {
sudo apt-get -qq update
local -a packs
packs=(build-essential debian-goodies libgccjit-10-dev)
sudo apt-get install $packs
}

You don't even need a configuration file nor a script to parse
it and pipe it to APT commands ...

It can be boiled down to nothing - however, the nothing works
- try the above command if you want a system that has
build-essential, debian-goodies and libgccjit-10-dev!

> Chef uses ruby with adaptations.
>
> Puppet uses a DSL written in ruby.
>
> ansible is entirely declarative, in YAML

Well, obviously all software is written in some language, if
the software is a new language tho for this purpose to pair
packages with configuration files that sounds like overkill to
me in one sense, and a disadvantage in another sense since one
would loose the features compared to just use a real languages
(with specific modules etc) like, again Elisp for Emacs and
Lua for mpv and some other software as well for that
matter ...

[Dan Ritter]

Emanuel Berg wrote:
> Dan Ritter wrote:
>
> >>> That's just knowing what packages you want to install.
> >>
> >> What do you mean, what else are you supposed to know?
> >
> > Examples:
>

> Okay, right, no here we're only concerned with the state of
> the OS in terms of packages that are installed so that they
> can be used immediately by the user, I know that all software
> can be and is by some including me configured with no end in
> sight but I see no reason to bring _that_ into _this_, since
> that is already done with config files and that's the best way
> there is to do it IMO, and now it's here as well - well, not
> really, that's what I'm asking for - but when it is for me as
> well, that'd be then end of it for me and I see no reason to
> mix it all together, plus there are other ways to automate
> bringing a bunch of files together on a disk if it comes
> to that.

So, that is the difference between package installation and
a configuration management system. As long as all you want is
package installation, you have a dozen ways to do it and you
clearly know four or five of them.

Chef, Puppet, ansible, and so forth are configuration management
systems.

-dsr-

[Andy Smith]

Hello,

On Mon, May 30, 2022 at 08:05:28AM -0400, Dan Ritter wrote:
> For a single user's machine, it's unlikely to be rewarding
> except intellectually.

It is however a great way to document a system for those that don't
get around to making free text notes. The language of the
configuration management tool both does the setup and documents what
needs to be done.

This is useful even for a single human, but if you have multiple
people working on things then there is some value in all of them
learning how to read the domain-specific language of the chosen
config management tool (e.g. Ansible, Puppet, etc.) vs. everyone
making their own notes in their own style.

That covers the "what"; generally more documentation is needed for
the "why", but even if it never comes the things in the config
management are better than nothing.

There have honestly been times in my life where I've had to look at
something set up by someone who's no longer around (or by me, a
decade previous!) and with only the things in config management I've
been able to work out which pieces the service is composed of, and
just that's been a God send and a bigger win than the automated
nature of the setup which is the thing people usually praise config
management for!

Cheers,
Andy

--
https://bitfolk.com/ -- No-nonsense VPS hosting

[Emanuel Berg]

Dan Ritter wrote:

>>>>> That's just knowing what packages you want to install.
>>>>
>>>> What do you mean, what else are you supposed to know?
>>>

>>> Examples [...]

>>
>> Okay, right, no here we're only concerned with the state of
>> the OS in terms of packages that are installed so that they
>> can be used immediately by the user, I know that all
>> software can be and is by some including me configured with
>> no end in sight but I see no reason to bring _that_ into
>> _this_, since that is already done with config files and
>> that's the best way there is to do it IMO, and now it's
>> here as well - well, not really, that's what I'm asking for

>> - but when it is for me as well, that'd be the end of it

>> for me and I see no reason to mix it all together, plus
>> there are other ways to automate bringing a bunch of files
>> together on a disk if it comes to that.
>
> So, that is the difference between package installation and
> a configuration management system. As long as all you want
> is package installation, you have a dozen ways to do it and
> you clearly know four or five of them.
>
> Chef, Puppet, ansible, and so forth are configuration
> management systems.

OK, thanks, that's a good definition ...

[Emanuel Berg]

What? :)

Except for people aging and dying I didn't understand any of
this post ...

[Dan Ritter]

Andy Smith wrote:
> On Mon, May 30, 2022 at 08:05:28AM -0400, Dan Ritter wrote:
> > For a single user's machine, it's unlikely to be rewarding
> > except intellectually.

[good arguments snipped]

> There have honestly been times in my life where I've had to look at
> something set up by someone who's no longer around (or by me, a
> decade previous!) and with only the things in config management I've
> been able to work out which pieces the service is composed of, and
> just that's been a God send and a bigger win than the automated
> nature of the setup which is the thing people usually praise config
> management for!

Yes but: all of these are no longer in the domain of a single
person's system, but rather concern a second person (or third or
Nth) trying to understand that system.

I suppose having that wouldn't be bad to remind your future self
what you had done, mind you. There are projects at work which I
don't touch for years... but then, that's where I do spend the
time to use configuration management.

-dsr-

[Andy Smith]

Hello,

On Thu, Jun 02, 2022 at 02:46:49AM +0200, Emanuel Berg wrote:
> Except for people aging and dying I didn't understand any of
> this post ...

I'm happy to explain any part of it that you were not able to
understand, but you'll have to be more specific.

I must have been very unclear indeed if all you got from it was
something about people ageing and dying as I don't think I wrote
anything about that.

[Emanuel Berg]

Andy Smith wrote:

>> Except for people aging and dying I didn't understand any
>> of this post ...
>
> I'm happy to explain any part of it that you were not able
> to understand, but you'll have to be more specific.

No, I understand now, as a blueprint for a human to read as
well, not just parsable by a computer.

Sure! My config files are very educative at least to me, often
when I forget something I make a search and find something
I wrote or something I can change just a little bit ...

OTOH is it like saying the advantage with C++ isn't just the
speed of the compiled binary, it is also that C++ programmers
can read and understand the code?