Package update and version comparisons experiments

[Marco Marongiu]

Hi there

I've been doing experiments today with CFEngine 3.4.4, package version
comparisons and package upgrades on a Debian system. I tried to write
wrappers around dpkg --compare-versions to fix the version numbers
before they are passed to the actual commands. I have given up after
messing with my own laptop a couple of times, but I'll give it another
try on a VM later. I got some interesting findings though, and some
questions.

First, the commands specified in package_version_less_command,
package_version_equal_command are invoked in a shell, and since it's
quite common for CFEngine to pass a "*" as an argument, this results in
passing to the command the current version, and the list of
files/directories in the root... ooops!!! Unless there is a good reason
to do so, which I'd like someone to explain, I believe that those
commands shouldn't run in a shell.

Second, when the policy is update (or addupdate for that matter) it
really makes no sense that CFEngine passes a "*" as an argument. Since
it knows both the installed version and the available version, it should
pass those two, plain and straight, for comparison. If I am wrong, can
someone please explain the logic for not doing so?

A final question: is any of these misbehaviours fixed in 3.5.x or 3.6?

thanks, ciao!
-- bronto

[Ted Zlatanov]

On Sat, 12 Apr 2014 22:39:36 +0200 Marco Marongiu <bront...@gmail.com> wrote:

MM> First, the commands specified in package_version_less_command,
MM> package_version_equal_command are invoked in a shell, and since it's
MM> quite common for CFEngine to pass a "*" as an argument, this results in
MM> passing to the command the current version, and the list of
MM> files/directories in the root... ooops!!! Unless there is a good reason
MM> to do so, which I'd like someone to explain, I believe that those
MM> commands shouldn't run in a shell.

I agree... this is the compounding of several internal issues.

MM> Second, when the policy is update (or addupdate for that matter) it
MM> really makes no sense that CFEngine passes a "*" as an argument. Since
MM> it knows both the installed version and the available version, it should
MM> pass those two, plain and straight, for comparison. If I am wrong, can
MM> someone please explain the logic for not doing so?

I also agree here: with the right `package_name_convention` this causes
CFEngine to install `package-*` which, in Yum-land, is not good at all.

MM> A final question: is any of these misbehaviours fixed in 3.5.x or 3.6?

I've tried to provide decent solutions in 3.6, which should work in 3.5
as well, with the `package_present`, `package_absent`, and
`package_latest` bundles. They are relatively new but I've tested them
with APT, Yum, and zypper. They don't try to install `package-*` and do
the right thing in general, I believe. Are they affected by the shell
behavior of `package_version_less_command` and
`package_version_equal_command`? I need a test case to verify, if you
have one.

Thanks
Ted

[Ted Zlatanov]

I wanted to follow up on packages promises in general, from my
perspective (please understand I'm not managing that area of the
CFEngine product, just a contributor).

There was a large internal effort at CFEngine to improve them over the
last few months. There are many new acceptance tests and checks on the
existing code.

(Definitions: file-based package repositories are a directory with
packages in it, possibly multiple versions. Normal package repositories
are remote and only accessible through a package manager such as Yum or
APT.)

I spent many hours documenting the functions in `verify_packages.c` over
the last few weeks. The inline C documentation now has a bit more detail
on why, for instance, `*` is used as the version number, and why `<`
doesn't work in some circumstances. Like other programmers, I wasn't
able to untangle that complexity fully while still passing all the
acceptance tests. The dissonance between file-based package repositories
and normal package repositories is particularly strong. You can look at
the comments yourself and make up your mind:
https://github.com/cfengine/core/blob/master/cf-agent/verify_packages.c

So, on my own, I've put up a proposal to just call package managers
directly with the right parameters. Take a look at
https://github.com/cfengine/masterfiles/pull/203 . I will write a
detailed blog entry on it but would appreciate your review and input.
It's a different direction from standard packages promises, based on
things I've done on my own over the years, and only covers Red Hat,
Debian, and SUSE derivatives. It will definitely not work for file-based
package repositories, but perhaps those should be entirely separate
because they differ in almost every way from normal package
repositories.

Thanks
Ted

[Marco Marongiu]

On 04/14/2014 12:11 PM, Ted Zlatanov wrote:
> On Sat, 12 Apr 2014 22:39:36 +0200 Marco Marongiu <bront...@gmail.com> wrote:
>
> MM> First, the commands specified in package_version_less_command,
> MM> package_version_equal_command are invoked in a shell

[...]
> MM> I believe that those

> MM> commands shouldn't run in a shell.
>
> I agree... this is the compounding of several internal issues.

In the meanwhile, the workaround is to put single quotes around the
variables in the package_version_less_command,
package_version_equal_command definitions. The commands will still be
run in a shell, but at least meta-characters won't be expanded.

> MM> A final question: is any of these misbehaviours fixed in 3.5.x or 3.6?
>
> I've tried to provide decent solutions in 3.6, which should work in 3.5
> as well, with the `package_present`, `package_absent`, and

> `package_latest` bundles. [...] Are they affected by the shell

> behavior of `package_version_less_command` and
> `package_version_equal_command`? I need a test case to verify, if you
> have one.

Is my test package, included in the email "dpkg version comparison
wrappers" good enough?

Ciao
-- bronto

[Marco Marongiu]

On 04/14/2014 12:39 PM, Ted Zlatanov wrote:
> So, on my own, I've put up a proposal to just call package managers
> directly with the right parameters.

I have checked your pull request (but not tested it).

I am OK to delegate more to the package managers in general, but I
believe that the agent should have enough intelligence to "filter"
requests, to understand if it's the case to call the package manager or
not, rather than blindly rely on if_elapsed to save resources.

The problem I see with if_elapsed is that, for example, in some cases
you want a package upgrade to take place immediately, at the first run
of the agent after the policy change (think of a critical bug as
heartbleed...). If I have to wait on if_elapsed, it's bad.

Of course I can work around if_elapsed by, for example, adding a new
promise (instead of adding to the list of the packages that I want to
always be at the latest version). I am happy to have a workaround and
bend CFEngine to my needs, but wouldn't it be much better if the
functionality did what it is supposed to?

I have an idea buzzing in my head since days, for which I haven't
sketched an implementation. If it was doable, it would also match (and
improve) your proposal. Rather than conditioning the package
installations/removals/upgrades to the same bounds as the package list
update, it would probably make sense to have the package list updates
save a cache in a "neutral" format (like it happens now), and have that
cache checked at every run by "your" package promises to decide if it's
the case or not to call the package manager, and get rid of if_elapsed.
From the top of my head, the package cache should include:

* package name (of course)
* installed version (or a marker like - if it's not installed)
* candidate version (version that the PM would install on upgrade)
* architecture

Notice that the candidate version is not necessarily the latest/highest
version available in the repositories. E.g., on my system I have:

> bronto@murray:~$ apt-cache policy gnome-shell
> gnome-shell:
> Installed: 3.8.4-5
> Candidate: 3.8.4-5
> Version table:
> 3.8.4-8 0
> -1 http://http.debian.net/debian/ jessie/main amd64 Packages
> *** 3.8.4-5 0
> 700 http://debian.lth.se/lmde/latest/ testing/main amd64 Packages
> 100 /var/lib/dpkg/status
> 3.4.2-7+deb7u1 0
> 1 http://http.debian.net/debian/ wheezy/main amd64 Packages

As you can see, the versions available for gnome-shell are
3.4.2-7+deb7u1, 3.8.4-5 and 3.8.4-8. However, due to APT pins, 3.8.4-8
is not the candidate. In this case, I would expect the cache to contain
something like

gnome-shell,3.8.4-5,3.8.4-5,amd64

rather than

gnome-shell,3.8.4-5,3.8.4-8,amd64

On the other hand, this is exactly what the agent should be doing
already, but it doesn't. So I am really wondering if I am re-inventing
the wheel rather than inventing a better one... Does this all make sense?

Ciao!
-- bronto

PS: I am not turning down the efforts of you guys to make this part of
CFEngine better, I don't mean to in any way. It's just that managing
packages is such a fundamental part of managing systems that it hurts
when it doesn't work the way we'd all want and expect and like.

[Ted Zlatanov]

On Mon, 14 Apr 2014 14:29:57 +0200 Marco Marongiu <bront...@gmail.com> wrote:

MM> In the meanwhile, the workaround is to put single quotes around the
MM> variables in the package_version_less_command,
MM> package_version_equal_command definitions. The commands will still be
MM> run in a shell, but at least meta-characters won't be expanded.

Like this?

https://github.com/cfengine/masterfiles/pull/204

Ted

[Ted Zlatanov]

On Mon, 14 Apr 2014 15:01:14 +0200 Marco Marongiu <bront...@gmail.com> wrote:

MM> I have an idea buzzing in my head since days, for which I haven't
MM> sketched an implementation. If it was doable, it would also match (and
MM> improve) your proposal. Rather than conditioning the package
MM> installations/removals/upgrades to the same bounds as the package list
MM> update, it would probably make sense to have the package list updates
MM> save a cache in a "neutral" format (like it happens now), and have that
MM> cache checked at every run by "your" package promises to decide if it's
MM> the case or not to call the package manager, and get rid of if_elapsed.

...

MM> Notice that the candidate version is not necessarily the latest/highest
MM> version available in the repositories. E.g., on my system I have:
MM> On the other hand, this is exactly what the agent should be doing
MM> already, but it doesn't. So I am really wondering if I am re-inventing
MM> the wheel rather than inventing a better one... Does this all make sense?

I think you're restating the current packages promises and trying to
override the package manager's worldview. That seems to be a difficult
task due to the variety and creativity of the package manager implementations.

I would rather manage each platform the way its package manager expects,
only stepping in for file-based package repositories (e.g. Solaris)
where the native package manager is too weak.

Ted

[Marco Marongiu]

On 04/14/2014 04:11 PM, Ted Zlatanov wrote:
> I think you're restating the current packages promises and trying to
> override the package manager's worldview.

That's not what I want. I want to bring the package manager world view
into CFEngine, so that I know when to call the package manager and when
it's OK to let it be.

> That seems to be a difficult task due to the variety and creativity
> of the package manager implementations.

Never thought it's easy. In fact, it's just something buzzing in my head
with no implementation (so far...)


Ciao!
-- bronto

[Marco Marongiu]

yep, like that.

Ciao
-- M

[Ted Zlatanov]

On Mon, 14 Apr 2014 19:46:44 +0200 Marco Marongiu <bront...@gmail.com> wrote:

MM> yep, like that.

Merged. Please comment in https://cfengine.com/dev/issues/5470 if you
have any issues with it.

Thanks
Ted

[Ted Zlatanov]

On Mon, 14 Apr 2014 19:45:49 +0200 Marco Marongiu <bront...@gmail.com> wrote:

MM> On 04/14/2014 04:11 PM, Ted Zlatanov wrote:
>> I think you're restating the current packages promises and trying to
>> override the package manager's worldview.

MM> That's not what I want. I want to bring the package manager world view
MM> into CFEngine, so that I know when to call the package manager and when
MM> it's OK to let it be.

OK, I think there's merit to that approach, but the file-based caching
you mentioned is already taking package manager concepts (what's a
package? a dependency? a version? etc.) and translating them. Yum and
APT, for instance, are fundamentally different in how they get and cache
updates to the available packages. I look forward to hearing about how
you and others want package management to look and work.

Ted