Patching and dealing with the consequences (restart / reboot conditions)

[Martin Simons]

Dear CFEngineer,

The CFEngine Roadshow is moving on and is reaching an interesting point. Customer request: Paniqing Big Bank board member manager, twoo sheets of vulnarabilities of unpatched issues in one of his applications.

Good idea to let CFEngine do the patching, a bit like this:
https://github.com/Webhuis/CFEngine-Roadshow/blob/cfgmgmtcamp2015/policies/dynamic_policies/platform/debian_7/debian.cf

The agent detects a mirror change and then does an upgrade, this is debian so apt-get upgrade. And suroise, surprise many lines show a line like this:
1424377627|I|409|10.168.0.11|security.debian.org/dists/wheezy/updates/main/binary-amd64/Packages.bz2

So, what if this patch requires a restart of an application or even a reboot?

I did not find a meachnism so far, which is key to the manager I mentioned above.

Regards,
Martin.

[Neil Watson]

Martin,

I think you should look at a combination of flag files and uptime
counts. Off the top of my head, determine your criteria for when a
reboot is needed.

reboot_for_patching::
"/usr/sbin/reboot";

Or

reboot_for_patching::
"touch /needs_reboot";

Or make the class persistent to make sure the reboot. Then clean up the
flag, in whatever form, at boot time. Use uptime to prevent extra
reboots.

--
Neil H Watson
Sr. Partner, Architecture and Infrastructure
CFEngine reporting: https://github.com/evolvethinking/delta_reporting
CFEngine policy: https://github.com/evolvethinking/evolve_cfengine_freelib
CFEngine and vim: https://github.com/neilhwatson/vim_cf3
CFEngine support: http://evolvethinking.com

[Nick Anderson]

Is newer than with pid 1.

And I agree with Neil.  First you must understand the specific conditions that indicate you need to reboot. Critical files that are newer than boot time might be a slightly loose but ok criteria.

Sometimes package managers have interesting query options. Maybe you could search recently installed packages for those that you know require a reboot.

Sent from my mobile device.

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengin...@googlegroups.com.
To post to this group, send email to help-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/help-cfengine.
For more options, visit https://groups.google.com/d/optout.

[Brian Bennett]

The checkrestart script in the debian-goodies package will search for any processes using outdated libs.

https://packages.debian.org/wheezy/all/debian-goodies/

I always used that. In the cf2 days I had classes that would be triggered after upgraded and look for processes that needed restart which would in turn trigger other classes that would restart the services.

Most Debian originated packages these days will auto restart when the package is upgraded. For private packages just make sure to add the proper restart commands to the postinst script (I don't remember the exact command, so check some other postinst scripts for the proper format).

-- 

Brian Bennett

Looking for CFEngine training?

http://www.verticalsysadmin.com/

--

[Martin Simons]

Dear Brian,

Picked this up again, after completing powerdns.

I have an old debian_7 bare image on which I install a Tomcat webserver. CFEngine carries out an apt-get upgrade and does a kernel ugrade:
security.debian.org/pool/updates/main/l/linux/linux-image-3.2.0-4-amd64_3.2.65-1+deb7u2_amd64.deb
Excellent for my demonstration, because this machine needs a reboot.
I saw an interesting message come by on console:
INTI: version 2.88 reloading
I doubt this is enough, in my opinion, although only the initrd.img is new.

This is checkrestart output:
Found 19 processes using old versions of upgraded files
(12 distinct programs)
(9 distinct packages)

Of these, 4 seem to contain init scripts which can be used to restart them:
The following packages seem to have init scripts that could be used
to restart them:
cfengine-community:
        2069    /var/cfengine/bin/cf-monitord
        2055    /var/cfengine/bin/cf-execd
        2062    /var/cfengine/bin/cf-serverd
acpid:
        1838    /usr/sbin/acpid
cron:
        1873    /usr/sbin/cron
udev:
        26236   /sbin/udevd
        26235   /sbin/udevd
        294     /sbin/udevd

These are the init scripts:
service cfengine3 restart
service acpid restart
service cron restart
service udev-mtab restart
service udev restart

These processes do not seem to have an associated init script to restart them:
dash:
        2121    /bin/dash
apt:
        2122    /usr/bin/apt-get
        2125    /usr/lib/apt/methods/http
        2126    /usr/lib/apt/methods/http
login:
        1918    /bin/login
bash:
        1927    /bin/bash

checkrestart seems to give no clue.

The old linux-image is being used:
Linux webapp0001 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux

After reboot it gives:
Linux webapp0001 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u2 x86_64 GNU/Linux
checkrestart output is empty, of course. :-)
Found 0 processes using old versions of upgraded files

Regards,
Martin.

Op vrijdag 20 februari 2015 03:25:06 UTC+1 schreef Brian Bennett:

[Dipayan Mallick]

Hi Martin

Following are the few situation to cite, where reboot of the Host is required:

1.       If a new/upgraded kernel is being installed in the Host.

      2.       If security patch being installed which requires a reboot (e.g glibc Ghost Vulnerabilities , etc.).

      3.       If there is an VMDK issue which results “/” partition to Read-Only.  

4.    If we need to patch SSL package and as many processes are linked to its library and we need a reboot.

 

Thanks,

 

ISD team

 

[Martin Simons]

Dear Dipayan,

In Debian even a successful dist-upgrade does not automatically lead to a reboot. :-)

So we're looking for a good policy to determine whether a new kernel or initrd.gz is in place, A change detecting files promise on /boot is useful here I feel.

How do you know or determine a security patch requires a reboot? Again a change detecting files promise (tripwire) on specific librarieswould do a good job here.

The ro / partition can be found in /proc I guess>

In case of the SSL patch, wouldn't a telinit 1 be sufficient?

This is an example of a change detecting promise, we could set a class here too IMHO:
bundle agent file_security {
files:
 "/etc"
  handle => "etc_tripwire",
  comment => "Report changes on files in /etc",
  changes => detect_all_change,
  depth_search => recurse("inf");
}

Doing this for /boot would be a bit tricky, because any change made to /boot would trigger a reboot if you set a restart_class. :-)

Best regards,
Martin.

Op maandag 30 maart 2015 16:01:45 UTC+2 schreef Dipayan Mallick:

[Nick Anderson]

On 03/31/2015 08:21 AM, Martin Simons wrote:
> So we're looking for a good policy to determine whether a new kernel or
> initrd.gz is in place, A change detecting files promise on /boot is useful
> here I feel.

You could use findfiles() to find all files in /boot.


Then you could iterate on all of those files, and see if any of them are
newer than pid 1.
https://docs.cfengine.com/latest/reference-functions-isnewerthan.html

```
"boot_files_changed_since_boot"
expression => isnewerthan("$(found_files), "/proc/1");

commands:
boot_files_changed_since_boot.automatic_reboot_allowed::
"/sbin/shutdown"
args => "-r now CFEngine initiated reboot, /boot related updates";
```

[Nick Anderson]

On Monday, March 30, 2015 at 9:01:45 AM UTC-5, Dipayan Mallick wrote:

Following are the few situation to cite, where reboot of the Host is required:

1.       If a new/upgraded kernel is being installed in the Host.

For this, I would monitor the files in /boot and see if they are newer than pid 1 (see my previous reply)

      2.       If security patch being installed which requires a reboot (e.g glibc Ghost Vulnerabilities , etc.).

Again, I would first try to monitor the critical files vs pid 1 to see if they are newer, which would indicate the need for a reboot. That might not work perfectly though. I suppose the times of the files in the package could be older than pid 1.

      3.       If there is an VMDK issue which results “/” partition to Read-Only. 

I am no vmware expert but usually when I had filesystems being mounted ro it was fundamentally because of slowly responding storage. When vmware tools is installed it increases the scsi timeout so that the system is more tolerant of slow storage (since its remote). If your frequently triggering this you will need to look for the root cause more closely.

Now as to how to detect this. There is inventory policy for mtab in masterfiles.
https://github.com/cfengine/masterfiles/blob/master/inventory/any.cf#L199

You could leverage it to see if / is mounted ro and then reboot.

4.    If we need to patch SSL package and as many processes are linked to its library and we need a reboot.

From here: http://askubuntu.com/questions/444702/how-to-patch-the-heartbleed-bug-cve-2014-0160-in-openssl I found this oneliner to list services using out of date ssl libraries. You could use something like this to detect the need for service restarts or a complete system reboot.

sudo find /proc -maxdepth 2 -name maps -exec grep -HE '/libssl\.so.* \(deleted\)' {} \; | cut -d/ -f3 | sort -u | xargs --no-run-if-empty ps uwwp

 

[Shane McEwan]

On 31/03/15 14:21, Martin Simons wrote:
> How do you know or determine a security patch requires a reboot?

Apparently[1], you can check for /var/run/reboot-required which is
created when a package indicates that it needs a reboot.

[1]
http://serverfault.com/questions/92932/how-does-ubuntu-keep-track-of-the-system-restart-required-flag-in-motd

Shane.

[Nick Anderson]

Nice, that one is new to me. What about the EL family?

[Neil Watson]

On Tue, Mar 31, 2015 at 03:47:58PM +0100, Shane McEwan wrote:
>On 31/03/15 14:21, Martin Simons wrote:
>Apparently[1], you can check for /var/run/reboot-required which is
>created when a package indicates that it needs a reboot.

I did not know this :). Trouble is, do I blindly trust this? Patching is
still very much a manual job. One has to test and verify before
considering a reboot. In which case the reboot command should be
dependent on the patching promise, rather than external files.

[Martin Simons]

Dear Shane,

You're my hero!

Is this Ubuntu only? Should we post this question *nix*?

I chaecked on of my Debian boxes and it is (not yet) there. I'll try another one that fo sure needs patching.

Best regards,
Martin.

Op dinsdag 31 maart 2015 16:48:05 UTC+2 schreef Shane McEwan:

[Shane McEwan]

One of the comments on the Serverfault link I posted in my reply states
that on Debian you need to have the update-notifier-common package
installed.

I don't know how/if other Unixen handle it.

Shane.

> <http://serverfault.com/questions/92932/how-does-ubuntu-keep-track-of-the-system-restart-required-flag-in-motd>
>
>
> Shane.

>
> --
> You received this message because you are subscribed to the Google
> Groups "help-cfengine" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to help-cfengin...@googlegroups.com

> <mailto:help-cfengin...@googlegroups.com>.

> To post to this group, send email to help-c...@googlegroups.com

> <mailto:help-c...@googlegroups.com>.

[Martin Simons]

Dear Shane,

Good tip!

I will add the update-notifier-common package to my debian-common policy!

Ted Zlatanov made a good remark IMHO on IRC, I saved the conversation but I lost it. He pointed to the fact that the condition for a reboot is independent  from the very reboot itself.

I would call this the modus in which a machine is operating. In an orchestration environment you would orchestrate the process of rebooting the machine. I feel this would add value to the mission portal and to community too of course.

I feel this is a very important feature from a business perspective. I work in a large organization where I see a CIO running around with two huge sheets A3 ( more than two letter sheets ) covered with red lines in small prints and telling about thousands of vulnerabilities her personally is responsible for.

I feel this is the case in a large enterprise where a reboot is a process that is carried out with care.
- Processes are being brought down in a controlled manner
- Backups are made
- It is an opportunity or a time window for other maintenance
- The machine is being brought down
- The machine boots
- Check are made

This way we offer the CIO a vehicle to be in control, again: very very important and a good USP if we manage to deliver.

Best regards,
Martin.

Op woensdag 1 april 2015 10:37:21 UTC+2 schreef Shane McEwan:

[Martin Simons]

Dear Shawn,

Could it be that update-notifier-common is replaced in Jessie?
https://github.com/opscode-cookbooks/apt/issues/80

Best regards,
Martin.

Op woensdag 1 april 2015 10:37:21 UTC+2 schreef Shane McEwan:

One of the comments on the Serverfault link I posted in my reply states

[Hans Spaans]

Hi Martin,

Maybe I overlooked it in the thread, but is the usecase to detect packages that require an update and a reboot afterwards, or to detect if an reboot is required?

Since Debian Wheezy and maybe even Squeeze some functionalities have been merged into apt itself.

Hans

Op donderdag 2 april 2015 14:09:16 UTC+2 schreef Martin Simons:

[Martin Simons]

Dear CFEngineer,

Happy New Year to you all!

There is something, on Debian at least ( Debian Rules). When do you want to be informed about an occurring reboot condition? When you security patch a piece of software that implies a reboot.

In the Debian world there is a special repository for security patching, meaning a security fix and non feature change for a given package. Because there are no new features involved the system should perform as desired after the patch. If not, which could of course happen in very rare circumstances, you still could argue that a broken system is the better option compared to a cracked system.

If you install the unattended-upgrade package, you will find the file:

/var/run/reboot-required

Good documentation:

https://wiki.debian.org/UnattendedUpgrades

http://wiki.dailystuff.nl/wiki/APT#Automatic_upgrades

In the latter you find the reference to the reboot required file. It would require the simplest of policies to check the contents of this file or to put a monitor on this file.

Why is unattended-upgrade so good?

- You automatically refresh your packages cache, like an apt-get update

- You may want to blacklist packages like kernel and libc

- You patch the basic platform stuff

- CFEngine already manages specified packages

- Application packages should have the reboot built in, if required.

Unattended-upgrade is configured within apt and there automated as such.

It will be in my standard set up as of now.

Regards,

Martin.

Op maandag 1 juni 2015 12:48:44 UTC+2 schreef Hans Spaans:

[Nick Anderson]

Martin Simons writes:

> If you install the unattended-upgrade package, you will find the file:
> /var/run/reboot-required
>
> Good documentation:
> https://wiki.debian.org/UnattendedUpgrades
> http://wiki.dailystuff.nl/wiki/APT#Automatic_upgrades

Nice find

> - Application packages should have the reboot built in, if required.

You mean that the application packages should be signaling during
postinstall that they need a reboot?

Seems like a package should be executing notify-reboot-required in
order to signal that /var/run/reboot-required should be created.

–
Nick Anderson
Doer of things, CFEngine

[Hans Spaans]

Op dinsdag 3 januari 2017 19:19:08 UTC+1 schreef Nick Anderson:

Martin Simons writes:

> If you install the unattended-upgrade package, you will find the file:
> /var/run/reboot-required
>
> Good documentation:
> https://wiki.debian.org/UnattendedUpgrades
> http://wiki.dailystuff.nl/wiki/APT#Automatic_upgrades

Nice find

> - Application packages should have the reboot built in, if required.

You mean that the application packages should be signaling during
postinstall that they need a reboot?

Correct and the kernel package should be an example of this. Other packages like openssh-servers restart the service themselves, but it may be wise to tell unattended-upgrades to do reduce downtime of a service by doing the update in small steps. You can use the following option:

Unattended-Upgrade::MinimalSteps "true"; 

 

Seems like a package should be executing notify-reboot-required in
order to signal that /var/run/reboot-required should be created.

Correct and other tools like Ansible use that one as trigger. This leaves the glibc update for example as an interesting corner case. In some cases Debian tries to restart running services, but in other cases it doesn't. I haven't seen a sane and safe way on how this should be handled except for letting a monitoring agent scan for a mismatch in linked libs on disk and for running processes. For now I have blacklisted some crucial libs for automated updates and update them in another way.

-- 

Hans

[Martin Simons]

Dear Nick,

Op dinsdag 3 januari 2017 19:19:08 UTC+1 schreef Nick Anderson:

Nice find

Thank you. I hope we will find equivalents for other platforms too. 

 

> - Application packages should have the reboot built in, if required.

You mean that the application packages should be signaling during
postinstall that they need a reboot?

I do not know so far how unattended-upgrade deals with all sorts of non standard packages, well written or not.

There is a wilderness of different kinds of packages out there.

- Backports

- Commercial software

- Home made software

The point is that alerts deal with the set of standard packages, many patches to packages in the list above will never make it to the Debian security repository. The more specific a package is the more unlikely it gets that exploits exist. Most of these packages exist under the radar of the Debian security team.

 

Seems like a package should be executing notify-reboot-required in
order to signal that /var/run/reboot-required should be created.

/var/run/reboot-required is managed by unattended-upgrade, so I feel other security upgrade processes for specific situations should leave it and devise their own mechanism.

Regards,

Martin.

 

[Martin Simons]

Dear Nick,

There is at least something available in Redhat, not very advanced though:

http://www.anchor.com.au/hosting/dedicated/Automatic-updates-for-servers

https://linuxaria.com/howto/enabling-automatic-updates-in-centos-7-and-rhel-7

yum-cron seems to be configurable for security upgrades only.

In the company I worked for they were reluctant to use automated security-upgrades, because in the past they broke the front-end webservers.

Regards,

Martin.

Op dinsdag 3 januari 2017 19:19:08 UTC+1 schreef Nick Anderson:

Martin Simons writes:

[Christian Linden]

There's a security plugin for yum:

https://www.cyberciti.biz/faq/redhat-fedora-centos-linux-yum-installs-security-updates/

c

[Nick Anderson]

Perhaps you would want to add this kind of information to the wiki that Steven started.

https://github.com/cfengine/core/wiki

--
You received this message because you are subscribed to a topic in the Google Groups "help-cfengine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/help-cfengine/NAEcvgPZlVw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to help-cfengine+unsubscribe@googlegroups.com.

To post to this group, send email to help-c...@googlegroups.com.

Visit this group at https://groups.google.com/group/help-cfengine.

[Christian Linden]

I miss the point "How to use CFEngine" =)

[Marco Marongiu]

On 05/01/17 18:40, Christian Linden wrote:
> I miss the point "How to use CFEngine" =)

That is covered here:

http://shop.oreilly.com/product/0636920022022.do?sortby=publicationDate

:-D

[Christian Linden]

That’s out of date.. lots of things are improved.

For me, honestly, who tried to learn CFEngine using this book, it was too complicated.
Now, 20 months later, looking in the book, I see the things clear. But I didn’t learn from there.
I learned from the simplest examples.

.. but I may take a another look…

c

[Diego Zamboni]

> That’s out of date.. lots of things are improved.

Indeed, the book covers CFEngine 3.5, so it’s quite an old version, many many things have improved. Sadly I haven’t had a chance for a long time to work on producing a new edition. Still, if you’ve read it and have any feedback about how it could be made easier for a beginner to learn from it, I’d be happy to hear it :)

Cheers,
—Diego

>
> For me, honestly, who tried to learn CFEngine using this book, it was too complicated.
> Now, 20 months later, looking in the book, I see the things clear. But I didn’t learn from there.
> I learned from the simplest examples.
>
> .. but I may take a another look…
>
> c
>

> --

> You received this message because you are subscribed to the Google Groups "help-cfengine" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengin...@googlegroups.com.

[Christian Linden]

Thanks for confirming, Diego =)

Writing HowTos, articles and emails take a long time.. but books.. EVIL!!!
I like to know how Mark B. is doing that.. as his kind of books are kind of unmeasurable effort..

I will have ideas how to make it easier, because I’m not a technical person and I have to deal with technical and even coding stuff all my life and it seems to be my fate. And even guys like me know after >15 years dealing with that kind of information how to deal with it =)
From this perspective I can explain how to deploy policies with what sense for what purpose to whomever.
Otherwise I may bore technical professionals. That’s the problem when addressing a book… or.. when addressing information =)

I’d like to write a German CFEngine book.. but I know about the time it takes.. and that’s the ideas end because I’m a poor donkey and I’ve to work for ugly money… =(

Your new edition will be easier and kind of easier to understand because a lot of shell commands became obsolete due to new promise types =)
This makes CFEngine more attractive to devops those are not too familiar (and don’t want to become) with bash, sed, etc.


Cheers,
Chris

[Marco Marongiu]

On 07/01/17 23:29, Diego Zamboni wrote:
> Indeed, the book covers CFEngine 3.5, so it’s quite an old version,
> many many things have improved. Sadly I haven’t had a chance for a
> long time to work on producing a new edition. Still, if you’ve read
> it and have any feedback about how it could be made easier for a
> beginner to learn from it, I’d be happy to hear it :)

I'll throw a crazy idea here.

I strongly disagree with Christian: to me, the real work on CFEngine
started when I read Diego's book. And it's a pity to see such a
masterpiece falling in obsolescence.

It's quite evident that the task is, at the moment, too big for Diego to
handle. But what if we started a collective effort, where one or more
people take care of just one chapter?

Ciao
-- bronto

[Diego Zamboni]

Hi Marco,

Thanks for the idea. I’ve had something like this in mind for a while, but I am still in the process of clearing up the legal/rights side with O’Reilly.

Best regards,
—Diego

[Aleksey Tsalolikhin]

Just a reminder, lots of simple examples in www.cfenginetutorial.org

:)

It is meant to complement Diego's book and the vendor documentation. 

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengine+unsubscribe@googlegroups.com.

[Diego Zamboni]

Thanks Aleksey - good work on the CFEngine Tutorial. I hadn't seen it before in this form, very impressive :)

--Diego