Bug#905772: virtlogd dependency loop cuaing upgrade failures

[Christian Ehrhardt]

Package: libvirt

Version: 4.6.0-1

Severity: normal

Hi,

I ran into this while testing libvirt 4.6 and opened initially just Ubuntu bug [1].

But I found nothing Ubuntu-special so I recreated on Debian and realized that you are affected as well.

I have no solution yet, but would be happy if we can sync on ideas and implement the same eventually whatever we come up with.

But first let me outline the issue

TL;DR:

- virtlogd.service has since 4.2 a requires statement to both sockets (virtlods.socket and virtlogd-admin.socket)

- before an upgrade usually virtlogd.socket and virtlogd.service are running

- on the upgrade virtlogd-admin.socket gets installed

Note: all those services/sockets run with no-restart-on-upgrade

- virtlogd should not be restarted, so it calls systemctl reload

- dh_systemd_enable: the two sockets files get enabled

- dh_systemd_start: will only call deb-systemd-invoke start on the sockets ignoring errors

- /etc/init.d/virtlogd exists, so the postinst calls "invoke-rc.d virtlogd start"

Remember that virtlogd is already running and working fine.

But that last step makes it realize that the new requires dependency to virtlogd-admin.socket is not fulfilled and aborts the installation.

In a similar fashion /etc/init.d/libvirtd always gives an extra potentially fatal "start" action.

Since the latter is not instaklled with no-restart dh_installinit will even make this a restart call on upgrades.

As a POC I took away:
  $ mv  /etc/init.d/virtlogd /etc/init.d/virtlogd.nothere
But then realized we still start libvirtd through "invoke-rc.d libvirtd $_dh_action" which due to this line in the init script:
  # Required-Start:    $network $local_fs $remote_fs $syslog virtlogd
will fail to start now.

Overall this is preferred by dh_systemd_start since both scripts are installed.

I think it might be time to drop both sysV scripts as it seems that would resolve all of it by eliminating the double calls.

But I need to test that without messing in a container, but a real package build to be sure.

Details of such an upgrade - example is stable 3.0 -> testing 4.5

PRE
root@debian-stretch:~# systemctl status virtlogd.service virtlogd.socket virtlogd-admin.socket virtlockd.service virtlockd.socket virtlockd-admin.socket --no-pager --lines 2
● virtlogd.service - Virtual machine log manager
   Loaded: loaded (/lib/systemd/system/virtlogd.service; indirect; vendor preset: enabled)
   Active: active (running) since Thu 2018-08-09 09:28:33 UTC; 7s ago
     Docs: man:virtlogd(8)
           http://libvirt.org
 Main PID: 7080 (virtlogd)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/virtlogd.service
           └─7080 /usr/sbin/virtlogd

Aug 09 09:28:35 debian-stretch systemd[1]: virtlogd.service: Failed to reset devices.list: Operation not permitted
Aug 09 09:28:37 debian-stretch systemd[1]: virtlogd.service: Failed to reset devices.list: Operation not permitted

● virtlogd.socket - Virtual machine log manager socket
   Loaded: loaded (/lib/systemd/system/virtlogd.socket; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-08-09 09:28:33 UTC; 7s ago
   Listen: /var/run/libvirt/virtlogd-sock (Stream)

Aug 09 09:28:33 debian-stretch systemd[1]: Listening on Virtual machine log manager socket.
Unit virtlogd-admin.socket could not be found.

● virtlockd.service - Virtual machine lock manager
   Loaded: loaded (/lib/systemd/system/virtlockd.service; indirect; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:virtlockd(8)
           http://libvirt.org

● virtlockd.socket - Virtual machine lock manager socket
   Loaded: loaded (/lib/systemd/system/virtlockd.socket; enabled; vendor preset: enabled)
   Active: active (listening) since Thu 2018-08-09 09:28:34 UTC; 7s ago
   Listen: /var/run/libvirt/virtlockd-sock (Stream)

Aug 09 09:28:34 debian-stretch systemd[1]: Listening on Virtual machine lock manager socket.
Unit virtlockd-admin.socket could not be found.

Upgrade just libvirt and dependencies - stretch to testing.

# apt install libvirt-daemon-system
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libboost-iostreams1.62.0 libboost-random1.62.0 libboost-system1.62.0 libboost-thread1.62.0 librados2 librbd1
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
  ca-certificates libcap-ng0 libcom-err2 libcomerr2 libcurl3-gnutls libidn2-0 libnghttp2-14 libpsl5 librtmp1 libunistring2 libvirt-clients
  libvirt-daemon libvirt0 openssl publicsuffix
Suggested packages:
  libvirt-daemon-driver-storage-gluster libvirt-daemon-driver-storage-rbd libvirt-daemon-driver-storage-sheepdog
  libvirt-daemon-driver-storage-zfs numad apparmor auditd nfs-common open-iscsi pm-utils radvd systemtap zfsutils
The following NEW packages will be installed:
  ca-certificates libcom-err2 libcurl3-gnutls libidn2-0 libnghttp2-14 libpsl5 librtmp1 libunistring2 openssl publicsuffix
The following packages will be upgraded:
  libcap-ng0 libcomerr2 libvirt-clients libvirt-daemon libvirt-daemon-system libvirt0
6 upgraded, 10 newly installed, 0 to remove and 248 not upgraded.
Need to get 9437 kB of archives.
After this operation, 5147 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
[...]
Setting up libvirt-daemon-system (4.5.0-1) ...
Installing new version of config file /etc/apparmor.d/abstractions/libvirt-qemu ...
Installing new version of config file /etc/apparmor.d/libvirt/TEMPLATE.lxc ...
Installing new version of config file /etc/apparmor.d/libvirt/TEMPLATE.qemu ...
Installing new version of config file /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper ...
Installing new version of config file /etc/apparmor.d/usr.sbin.libvirtd ...
Installing new version of config file /etc/default/libvirt-guests ...
Installing new version of config file /etc/init.d/libvirt-guests ...
Installing new version of config file /etc/libvirt/libvirtd.conf ...
Installing new version of config file /etc/libvirt/libxl.conf ...
Installing new version of config file /etc/libvirt/qemu.conf ...
Installing new version of config file /etc/libvirt/virtlockd.conf ...
Installing new version of config file /etc/libvirt/virtlogd.conf ...
Installing new version of config file /etc/logrotate.d/libvirtd.libxl ...
Installing new version of config file /etc/logrotate.d/libvirtd.lxc ...
Installing new version of config file /etc/logrotate.d/libvirtd.qemu ...
Installing new version of config file /etc/logrotate.d/libvirtd.uml ...
Installing new version of config file /etc/sasl2/libvirt.conf ...
Created symlink /etc/systemd/system/sockets.target.wants/virtlockd-admin.socket → /lib/systemd/system/virtlockd-admin.socket.
Created symlink /etc/systemd/system/sockets.target.wants/virtlogd-admin.socket → /lib/systemd/system/virtlogd-admin.socket.
virtlockd.service is a disabled or a static unit, not starting it.
Job for virtlogd-admin.socket failed.
See "systemctl status virtlogd-admin.socket" and "journalctl -xe" for details.
virtlogd-admin.socket couldn't start.
A dependency job for virtlogd.service failed. See 'journalctl -xe' for details.
invoke-rc.d: initscript virtlogd, action "start" failed.
● virtlogd.service - Virtual machine log manager
   Loaded: loaded (/lib/systemd/system/virtlogd.service; indirect; vendor preset: enabled)
   Active: active (running) since Thu 2018-08-09 09:35:36 UTC; 50s ago
     Docs: man:virtlogd(8)
           https://libvirt.org
 Main PID: 6774 (virtlogd)
   CGroup: /system.slice/virtlogd.service
           └─6774 /usr/sbin/virtlogd

Aug 09 09:35:38 debian-stretch systemd[1]: virtlogd.service: Failed to reset devices.list: Operation not permitted
Aug 09 09:35:41 debian-stretch systemd[1]: virtlogd.service: Failed to reset devices.list: Operation not permitted
Aug 09 09:36:21 debian-stretch systemd[1]: virtlogd.service: Failed to reset devices.list: Operation not permitted
Aug 09 09:36:25 debian-stretch systemd[1]: virtlogd.service: Failed to reset devices.list: Operation not permitted
Aug 09 09:36:25 debian-stretch systemd[1]: Reloading Virtual machine log manager.
Aug 09 09:36:25 debian-stretch systemd[1]: Reloaded Virtual machine log manager.
Aug 09 09:36:26 debian-stretch systemd[1]: virtlogd.service: Failed to reset devices.list: Operation not permitted
Aug 09 09:36:26 debian-stretch systemd[1]: virtlogd.service: Failed to reset devices.list: Operation not permitted
Aug 09 09:36:27 debian-stretch systemd[1]: Dependency failed for Virtual machine log manager.
Aug 09 09:36:27 debian-stretch systemd[1]: virtlogd.service: Job virtlogd.service/start failed with result 'dependency'.
dpkg: error processing package libvirt-daemon-system (--configure):
 subprocess installed post-installation script returned error exit status 1
Processing triggers for ca-certificates (20170717) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Processing triggers for libc-bin (2.24-11+deb9u3) ...
Processing triggers for systemd (232-25+deb9u4) ...
Errors were encountered while processing:
 libvirt-daemon-system
E: Sub-process /usr/bin/dpkg returned an error code (1)

Status after install:

# systemctl status virtlogd.service virtlogd.socket virtlogd-admin.socket virtlockd.service virtlockd.socket virtlockd-admin.socket --no-pager --lines 2
● virtlogd.service - Virtual machine log manager
   Loaded: loaded (/lib/systemd/system/virtlogd.service; indirect; vendor preset: enabled)
   Active: active (running) since Thu 2018-08-09 09:35:36 UTC; 1min 11s ago
     Docs: man:virtlogd(8)
           https://libvirt.org
 Main PID: 6774 (virtlogd)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/virtlogd.service
           └─6774 /usr/sbin/virtlogd

Aug 09 09:36:27 debian-stretch systemd[1]: virtlogd.service: Job virtlogd.service/start failed with result 'dependency'.
Aug 09 09:36:31 debian-stretch systemd[1]: virtlogd.service: Failed to reset devices.list: Operation not permitted

● virtlogd.socket - Virtual machine log manager socket
   Loaded: loaded (/lib/systemd/system/virtlogd.socket; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-08-09 09:35:36 UTC; 1min 11s ago
   Listen: /var/run/libvirt/virtlogd-sock (Stream)

Aug 09 09:35:36 debian-stretch systemd[1]: Listening on Virtual machine log manager socket.

● virtlogd-admin.socket - Virtual machine log manager socket
   Loaded: loaded (/lib/systemd/system/virtlogd-admin.socket; enabled; vendor preset: enabled)
   Active: inactive (dead)
   Listen: /var/run/libvirt/virtlogd-admin-sock (Stream)

Aug 09 09:36:27 debian-stretch systemd[1]: virtlogd-admin.socket: Socket service virtlogd.service already active, refusing.
Aug 09 09:36:27 debian-stretch systemd[1]: Failed to listen on Virtual machine log manager socket.

● virtlockd.service - Virtual machine lock manager
   Loaded: loaded (/lib/systemd/system/virtlockd.service; indirect; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:virtlockd(8)
           https://libvirt.org

● virtlockd.socket - Virtual machine lock manager socket
   Loaded: loaded (/lib/systemd/system/virtlockd.socket; enabled; vendor preset: enabled)
   Active: active (listening) since Thu 2018-08-09 09:36:26 UTC; 21s ago
   Listen: /var/run/libvirt/virtlockd-sock (Stream)

Aug 09 09:36:26 debian-stretch systemd[1]: Stopping Virtual machine lock manager socket.
Aug 09 09:36:26 debian-stretch systemd[1]: Listening on Virtual machine lock manager socket.

● virtlockd-admin.socket - Virtual machine lock manager admin socket
   Loaded: loaded (/lib/systemd/system/virtlockd-admin.socket; enabled; vendor preset: enabled)
   Active: active (listening) since Thu 2018-08-09 09:36:26 UTC; 21s ago
   Listen: /var/run/libvirt/virtlockd-admin-sock (Stream)

Aug 09 09:36:26 debian-stretch systemd[1]: Listening on Virtual machine lock manager admin socket.

[1]: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1786179

--

Christian Ehrhardt

Software Engineer, Ubuntu Server

Canonical Ltd

[Christian Ehrhardt]

Hi,

Just FYI simply removing the installation of the sysV scripts will uncover other issues.

If sysV and systemd services are installed the former still are prefered.

So with libvirtd sysV gone it will pick up libvirtd.service

That will parse the Requires which contains virtlogd-socket  will eventually lead to:

  deb-systemd-invoke $_dh_action 'libvirtd.service' 'virtlockd.socket' 'virtlogd.socket'

And that will stop the service to restart the socket.

I have taken a few more approcahes to this, like full conversion to dh_isntallsystemd and others.

But all failed so far by still calling the command above which will effectively restart virtlogd which is not what we want.

I start to consider things that look bad at first, for example keep the libvirtd sysV script but make it almost empty.

At least the dependencies to the virtlogd sysV script have to go away - otherwise it will block on that before the sysV to systemd wrapper kicks in :-/

Hoping you have a better Idea how to resolve,
Christian

[Christian Ehrhardt]

I set pkg-systemd...@lists.alioth.debian.org to CC on this as it is almost more a dh_*systemd* question/bug than of libvirt.

Libvirt just happens to be the package getting into this situation.

And their expertise might help to resolve this bug (#905772)

Trying an interim TL;DR:

- Service A - --no-stop-on-upgrade

  A Requires A1.socket

  A Requires A2.socket

- Service B - --restart-after-upgrade

  B Requires=A1.socket

- A and B have also sysV scripts.

- d/rules calls dh_installinit and dh_systemd_start

- maintscripts get:

  - invoke.rc start A - ok

  - invoke.rc restart B - ok

  - deb-systemd-invoke A.socket - this will restart A.service and breaks --no-stop-on-upgrade

This is confusing enough - re-summarize the approaches I had so far.

Remember:
- when there is a sysV script it will call invoke.rc
- without sysV script it will call deb-systemd-invoke

Original Issue:
- invoke.rc on virtlogd
  - this will realize the new dependency to virtlogd-admin.socket
  - virtlogd-admin.socket can't be started because

    virtlogd-admin.socket: Socket service virtlogd.service already active, refusing.

  - virtlogd.service is running fine, but the start returns RC!=0
  - that makes the upgrade fail

Fix I:
- drop both sysV scripts (virtlogd/libvirtd)
  - virtlogd will be started via deb-systemd-invoke
    This ignores errors in the postinst and is fine on upgrade
  - but libvirtd being taken over by deb-systemd-invoke is bad
    - the dh_systemd_start checks the service file and adds dependencies to the line
      deb-systemd-invoke restart 'libvirtd.service' 'virtlockd.socket' 'virtlogd.socket'
    - it knows that to restart a socket the service has to be stopped and started
    - so virtlogd is restarted ignoring the --no-restart-on-upgrade of the actual virtlogd
      service

Fix II:
- drop only virtlogd sysV script
  - virtlogd will be started via deb-systemd-invoke
    This ignores errors in the postinst and is fine on upgrade
  - libvirtd will continue to be started by invoke.rc which will restart "just" libvirtd
    (from dh_installinit)
  - but dh_systemd_start will have added a restart section
    deb-systemd-invoke $_dh_action 'virtlockd.socket' 'virtlogd.socket' >/dev/null || true
  - this will still restart the virtlogd service which has originally
    dh_installinit -p libvirt-daemon-system --name=virtlogd --no-restart-on-upgrade
  - We have two dh-systemd_start calls in d/rules, assuming retain order
    dh_systemd_start -p libvirt-daemon-system --restart-after-upgrade libvirtd.service
    is the one which makes it generate the restart on the sockets.
    It seems to realize that libvirtd will be started by invoke.rc, so it leaves that out, but
    will trigger the two dependencies.

Fix III:
- drop virtlogd sysV script and change the dh_systemds_start order
  - hope is that the dh_systemd_start of libvirtd considers the .socket
    files already handled and would no more add them with restart due to dependencies.
  - It changed the order in the generated maintainer script
  - But the section triggered by libvirtd.service still adds the sockets to the restart action
    deb-systemd-invoke $_dh_action 'virtlockd.socket' 'virtlogd.socket' >/dev/null || true
  - Please do mind, as in other cases here libvirtd itself is not here
    as it is taken over by the sysV start via invoke.rc

Fix IV:

  - a try to convert to compat 11 and onyl dh_installsystemd but failed
    at too many compat-11 implications

Fix V:
 - drop virtlogd sysV script (to fix the original issue) and drop the dh_systemd__start call to
   libvirtd (to avoid the secondary issue)
 - Intention: libvirtd (re)start is taken care of by dh_installinit anyway, avoid the bad restarts
   on virtlogd with this tweak
 - with that it seems to work, but it might have other implications that I missed
   - the new virtlogd-admin.socket is down (as it would need to restart the service)
   - service itself is up and still has the old PID so all is good
   - installation works, no more breaking the upgrade.

I'm not so sure if "Fix V" has other bad implications that I miss.
But currently that seems to work as needed - so feel free to consider [1] which is the code for it.

I think I'd want/need a "dh_systemd_start --no-dependent-services/sockets" option to intentionally have it generate "just" for libvirt.service and not the sockets it depends on.

As mentioned, for all of the complexity pulling in the systemd people might help as well.

So I'm eager to see what they will reply here as well.

[1]: https://git.launchpad.net/~libvirt-maintainers/ubuntu/+source/libvirt/commit/?h=ubuntu/cosmic-4.6&id=16bece58d55e50ca6d192714a3581ef981974288

[Michael Biebl]

Am 10.08.2018 um 13:32 schrieb Christian Ehrhardt:
> I think I'd want/need a "dh_systemd_start --no-dependent-services/sockets"
> option to intentionally have it generate "just" for libvirt.service and not
> the sockets it depends on.
> As mentioned, for all of the complexity pulling in the systemd people might
> help as well.
> So I'm eager to see what they will reply here as well.

I guess the complication arises from the fact that
dh_installinit/invoke-rc.d directly handles systemd service files if the
SysV init script and service file name match.
dh_installsystemd only handles those unit files for which no
corresponding SysV init script exists.

I think the solutions for this could be, to let dh_installsystemd handle
all systemd unit files.
dh_installinit/invoke-rc.d on the other hand would be updated to only
handle SysV init scripts.

In the long run I guess this will be less confusing at is clearer which
tool is responsible for which task and it makes it easier to override
the behaviour in debian/rules.

Felipe has been doing some initial work for enable that kind of
behaviour at
https://salsa.debian.org/debian/init-system-helpers/merge_requests/4

[Christian Ehrhardt]

On Fri, Aug 10, 2018 at 1:42 PM Michael Biebl <bi...@debian.org> wrote:

Am 10.08.2018 um 13:32 schrieb Christian Ehrhardt:
> I think I'd want/need a "dh_systemd_start --no-dependent-services/sockets"
> option to intentionally have it generate "just" for libvirt.service and not
> the sockets it depends on.
> As mentioned, for all of the complexity pulling in the systemd people might
> help as well.
> So I'm eager to see what they will reply here as well.


I guess the complication arises from the fact that
dh_installinit/invoke-rc.d directly handles systemd service files if the
SysV init script and service file name match.
dh_installsystemd only handles those unit files for which no
corresponding SysV init script exists. 

 

I think the solutions for this could be, to let dh_installsystemd handle
all systemd unit files.
dh_installinit/invoke-rc.d on the other hand would be updated to only
handle SysV init scripts.

In the long run I guess this will be less confusing at is clearer which
tool is responsible for which task and it makes it easier to override
the behaviour in debian/rules.

While the change of "responsibility" who is handling a service is puzzling at first, it is still rather clear.

I don't think changing it here would fix our current issue.

I appreciate these changes for the sake of clarity thou.

Consider what I wrote happens when I remove all sysV scripts.

The dh_*system code will generate the code with "restart" but not only for the listed service, but also required sockets it seems. 

The problem is the bleeding of -restart-after-upgrade into other elements that got set with --no-stop-on-upgrade.

In the current case that is libvirtd (-restart-after-upgrade) bleeding into the sockets listed as required.

In the example I added A is not to be restarted, but that is exactly what happens due to the "deb-invoke-systemd restart B.service A2.socket"

We'd either need logic to consider what those other elements were set to (restart or not) and pull required services in under those conditions.

Or we consider it a rare case anyway and would add automation code, but instead a switch which can disable pulling required services in.

Never the less all of the above might be only long term solutions.

Short term libvirt needs a way to restart "libvirtd" without restarting "virtlogd".

I have added my "Fix V" which would achieve that, but I'd appreciate:

- to hear (from systemd experts) if there might be other solutions than the ones already tried to achieve that?

- to hear from Guido about my Fix V being an interim solution for libvirt for now?

In fact since the former "dh_systemd_start libvirtd" ignored the service since there is a equally named sysV script - thereby we don't really loose anything here.

The only real thing that might need consideration is the removal of virtlogd sysV script.

It is required to fix the issue and not a problem for me on the Ubuntu side.

But I thought to remember Debian allows to switch init systems, so I'm not sure if that is acceptable for you?

Felipe has been doing some initial work for enable that kind of
behaviour at
https://salsa.debian.org/debian/init-system-helpers/merge_requests/4

[Felipe Sateler]

This should help, but then you get the next problem: dh_installsystemd parses the Also= lines, and generates deb-systemd-invoke for all referenced units.

I think the Also= lines in libvirtd.service are superfluous and removing them should avoid this problem. 

I'm still not sure why we parse Also= for starting. Michael, do you remember the rationale?

--

Saludos,
Felipe Sateler

[Christian Ehrhardt]

Uh I see, you mean dropping this section from libvirtd.service right?

[Install]
WantedBy=multi-user.target
Also=virtlockd.socket
Also=virtlogd.socket

The above is certainly worth a try - I need to check what I "loose" by that.

I'd still need to drop virtlogd sysV script as the "invoke.rc virtlogd" will complain about missing dependencies (the new .socket for it can't be started since the service is already running).

The dh_systemd_start generated code triggers "start" and ignores the retval, the dh_installinit code through invoke.rc calls start but fails since systemd replied "I'm running but there are dependency issues".

Because virtlogd.service has Requires virtlogd.socket and virtlogd-admin.socket.

You end like:

virtlogd.service - Virtual machine log manager
   Loaded: loaded (/lib/systemd/system/virtlogd.service; indirect; vendor preset: enabled)

   Active: active (running) since Thu 2018-08-09 05:26:07 UTC; 3h 45min ago

     Docs: man:virtlogd(8)
           https://libvirt.org

 Main PID: 4059 (virtlogd)

    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/virtlogd.service

           └─4059 /usr/sbin/virtlogd

Aug 09 09:10:37 c2 systemd[1]: Dependency failed for Virtual machine log manager.
Aug 09 09:10:37 c2 systemd[1]: virtlogd.service: Job virtlogd.service/start failed with result 'dependency'.

That has RC=1 if run directly.

And as I said, dh_systemd_start code will ignore the RC but dh_installinit won't

If there is also a good hint on that let us know.

 

--

Saludos,
Felipe Sateler

[Felipe Sateler]

I'd say nothing. libvirtd.service already Requires the sockets, so they will be pulled in anyway. 

Note that the Also= lines are actively harmful. If you want to disable automatic start at boot, and instead rely on socket activation, doing `systemctl disable libvirtd.service` will also disable the sockets.

 

I'd still need to drop virtlogd sysV script as the "invoke.rc virtlogd" will complain about missing dependencies (the new .socket for it can't be started since the service is already running).

The dh_systemd_start generated code triggers "start" and ignores the retval, the dh_installinit code through invoke.rc calls start but fails since systemd replied "I'm running but there are dependency issues".

Because virtlogd.service has Requires virtlogd.socket and virtlogd-admin.socket.

Yes, this is the part that could be addressed with the merge request Michael referenced.

--

Saludos,
Felipe Sateler

[Christian Ehrhardt]

On Fri, Aug 10, 2018 at 6:59 PM Michael Biebl <bi...@debian.org> wrote:

Am 10.08.2018 um 15:54 schrieb Felipe Sateler:
> I'm still not sure why we parse Also= for starting. Michael, do you
> remember the rationale?

I think this is simply a mistake and should be corrected.
Parsing Also= for dh_systemd_enable was done, so the tools behave
similar to how "systemctl enable foo" would behave.
I can't come up (or remember) a good idea why Also= would make sense for
the start case.

Hi,

thanks for all your participation in the discussion already.

For now I think we have several things to do, with rather different timelines.

1. Ubuntu-libvirt:

For me on the Ubuntu side since a release is rather close and we care less about sysV the immediate solution is to drop the sysV scripts and remove the Also= lines in the libvirtd.service.

I tested that and right now, it seems the only combination which makes the upgrade path work properly.

2. Debian-Libvirt:

I have made the change available as [1] for Guido to consider, although I'd understand if in Debian where sysV is still allowed to be switched to and no Feature Freeze nearby yet, you might wait until the changes in the dh_*systemd* tools become available.

@Guido Günther - it would be great if you'd let me know you preference/assumption for this, since IF you pick up (some of) this then I'd want to align myself.

If not we have to diverge for a while and get closer in both Distributions later on again.

3. dh_*systemd*

I assume you'll continue as usual on the already existing referenced merge request that would fix issue #1.

Probably one of you might also work on a new change to only parse the Also= for enable/disable but not start/restart actions.

I'm not sure about how to do that due to the ordering of the dh_ calls, but essentially if d/rules marked a service as --no-start-on-upgrade it should be considered by all calls.

So in our case, even if virtlogd services/sockets would be dragged in via Also= or any other, they should not inherit the --restart-after-upgrade from libvirt, but keep "their own" as specified.

Not sure how you'd keep the state across different calls thou, especially as the new dh_installsystemd is no more split in dh_installinit/dh_systemd_start, but I'd leave that up to you as you know the code much better.

If you spawn other bug(s) for any of that, I'd be interested to know so I can subscribe myself to them.

[1]: https://salsa.debian.org/libvirt-team/libvirt/merge_requests/5

[Sam Hartman]

control: severity -1 serious

justification: libvirtd upgrades from stretch to buster break causing
apt to fail and requiring the admin to get the systemd units into a
consistent state before things can continue


Unfortunately based on discussion so far this is a complex bug to fix.
Ubuntu's solution is to drop the sysv scripts and to drop Also= lines
in some of the units.

The systemd maintainers proposed that dropping Also as well as some
changes to move toward dh_systemd_start being used even when sysvinit
scripts are present would help this situation. Unfortunately it at
least doesn't look like those changes are in debhelper for buster.
Systemd folks, do you have any suggestions on how to approach this for
buster?

[Michael Biebl]

Hi Sam

Using debhelper compat level 12, you are able to completely decouple
dh_installinit and dh_installsystemd which would give you the ability to
implement what you want afaics.

Regards,
Michael


--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[Guido Günther]

Hi,

On Mon, Apr 15, 2019 at 10:18:18PM +0200, Michael Biebl wrote:
> Hi Sam
>
> Am 15.04.2019 um 20:38 schrieb Sam Hartman:
> > control: severity -1 serious
> >
> > justification: libvirtd upgrades from stretch to buster break causing
> > apt to fail and requiring the admin to get the systemd units into a
> > consistent state before things can continue
> >
> >
> > Unfortunately based on discussion so far this is a complex bug to fix.
> > Ubuntu's solution is to drop the sysv scripts and to drop Also= lines
> > in some of the units.
> >
> > The systemd maintainers proposed that dropping Also as well as some
> > changes to move toward dh_systemd_start being used even when sysvinit
> > scripts are present would help this situation. Unfortunately it at
> > least doesn't look like those changes are in debhelper for buster.
> > Systemd folks, do you have any suggestions on how to approach this for
> > buster?
>
> Using debhelper compat level 12, you are able to completely decouple
> dh_installinit and dh_installsystemd which would give you the ability to
> implement what you want afaics.

So let's move libvirt from 8 to 12 for stretch? I'm all for it but it'll
be a couple of days until I can set time aside for this.
cheers,
-- Guido

>
> Regards,
> Michael
>
>
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
>

> _______________________________________________
> Pkg-libvirt-maintainers mailing list
> Pkg-libvirt...@alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-libvirt-maintainers

[Sam Hartman]

Guido let me know if you need any help or prod me on IRC if I'm needed.
Will certainly test the result, but if you get stuck would be happy to
spend time on this.

[Guido Günther]

Hi,

On Mon, Apr 15, 2019 at 02:38:27PM -0400, Sam Hartman wrote:
> control: severity -1 serious
>
> justification: libvirtd upgrades from stretch to buster break causing
> apt to fail and requiring the admin to get the systemd units into a
> consistent state before things can continue
>
>
> Unfortunately based on discussion so far this is a complex bug to fix.
> Ubuntu's solution is to drop the sysv scripts and to drop Also= lines
> in some of the units.

Did you reproduce this bug on a stretch->buster upgrade? Cause I just
did that and didn't encounter any errors.

Cheers,
-- Guido

[Guido Günther]

Hi,
On Wed, Apr 17, 2019 at 02:47:41PM -0400, Sam Hartman wrote:
> but I haven't been able to reproduce on a clean install.
> It's frustrating: on my machines where I really want the upgrade to be
> smoothe this bit me, but on all my toy tests, it didn't happen.
> What I think may be necessary is for virtlogd to be active.
> So it may be necessary to actually get libvirtd running and actually
> running a VM to use the socket before the issue comes up.

That's possible.

> Alternatively, it's possible some other change has fixed this in the
> last month.

We did not change anything in that area so it's likely hidden (e.g. only
happening when running a VM) - I'll check that.


> I'll certainly say that a month ago ran into this on two different VM
> servers.

That's an important data point. Thanks
-- Guido

[Sam Hartman]

When I marked the bug RC, I thought it was happening all the time; I
then went to reproduce yet again in a controlled environment to be more
in a position to test a fix.
That's when I discovered things are more complex.

Obviously if this is a fringe situation, then it shouldn't be RC.

[Christian Ehrhardt]

Hi,
I checked this issue for Ubuntu bug 1786179 as I wanted to drop the
related delta that we formerly had. That is the same topic as
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=905772 that we
discuss here.

At first I thought the changes to dh_ resolved this sysV-vs-systemd
fight as I've seen it happen in other packages.

After initial install we have all sockets, but no service running => ok
But if services are running (later in the live-cycle) and we trigger a
reinstall/upgrade they are all restarted (which they are not supposed
to be done).

4 0 6927 6039 20 0 68836 55752 poll_s S+ ? 0:00
\_ apt install --reinstall libvirt-daemon-system
4 0 7045 6927 20 0 12924 4524 do_wai Ss+ pts/1 0:00
\_ /usr/bin/dpkg --status-fd 25 --configure --pending
0 0 7046 7045 20 0 25560 16328 do_wai S+ pts/1 0:00
\_ /usr/bin/perl -w /usr/share/debconf/frontend
/var/lib/dpkg/info/libvirt-daemon-system.postinst configure 5.4.
4 0 7055 7046 20 0 2572 1352 do_wai S+ pts/1 0:00
\_ /bin/sh
/var/lib/dpkg/info/libvirt-daemon-system.postinst configure
5.4.0-0ubuntu1~ppa6
0 0 7492 7055 20 0 11028 2464 poll_s S+ pts/1 0:00
\_ /bin/systemctl restart libvirt-guests.service
virtlockd-admin.socket virtlockd.service virtlockd.sock


This causes it to hang for 30-60 seconds on the upgrade which isn't
good but also not too bad.
But the logd/lockd services are restarted (I see new PIDs) which is
bad since those are all "--no-stop-on-upgrade".

I went back to (I know this is less of an option for Debian, but it
proved to be a great stop gap measure all too often) drop the sysV
scripts.

And back on that code not only does it look fine after install, now
also reinstall/upgrade work again.
The PIDs stay constant and no hang is perceived.
- install (sockets up, services down) = good
- reinstall (sockets up, services down) = good
- reinstall with services up
- no hang perceived = good
- PIDs still change = bad

So it is better without sysV, but not perfect (as in former versions) either.

Since there were discussions about reproducibility, the shortest test IMHO is:
$ apt install libvirt-daemon-system
$ systemctl start virtlogd.service
$ systemctl start virtlockd.service
$ systemctl status virtlogd.service virtlockd.service --no-pager
--lines 1 | grep PID
Main PID: 14021 (virtlogd)
Main PID: 14020 (virtlockd)
$ apt install --reinstall libvirt-daemon-system
# the PIDs will have changed, but they should not being "--no-stop-on-upgrade"

With compat 12 (even with sysV dropped as we did in Ubuntu) the
services will restart which is not what is wanted.
I verified e.g. Disco which has libvirt 5.0 and the sysV dropped, no
problem there.

For simplicity I compare/debug the versions with sysV dropped (less
interactions).
A lot of the postinst is the same, here "old = good" (Disco based on
libvirt 5.0) and "new = bad" (Eoan based on your 5.3 updated to
libvirt 5.4 from upstream).

old:
dh_systemd_start/12ubuntu1 for libvirtd.service
=> deb-systemd-invoke $_dh_action 'libvirtd.service' (action => restart)
dh_systemd_start/12ubuntu1 for "all others"
deb-systemd-invoke start ... (all services and sockets)
# on already running services that is a no-op the PIDs stay the same
new:
dh_installsystemd/12.1.1ubuntu1 for all services
deb-systemd-invoke $_dh_action (action => restart)
# this is what breaks the current libvirt, calling restart on the service

I started a buster system to check if this is special to Ubuntu, or at
least only to what we have in -experimental.
Ok, as initially reported on the bug here in Buster this issue applies
to virtlockd but not virtlogd.
I installed 5.2.0-2 from experimental, and there as expected both PIDs
are recycled.
Which makes sense given that it seems a non libvirt specific issue in
the postinst as generated by dh_installsystemd.

The d/rules line for all affected services surely has --no-stop-on-upgrade:
dh_installsystemd -p libvirt-daemon-system --no-stop-on-upgrade
$(LIBVIRT_SYSTEM_SERVICES)

Not sure what to do yet - I'm experimenting, but I wanted to keep you
in the loop as well.

[Christian Ehrhardt]

Was:
systemctl --system daemon-reload >/dev/null || true
if [ -n "$2" ]; then
_dh_action=restart
else
_dh_action=start
fi
deb-systemd-invoke $_dh_action 'libvirt-guests.service'
'virtlockd-admin.socket' 'virtlockd.service' 'virtlockd.socket'
'virtlogd-admin.socket' 'virtlogd.service' 'virtlogd.socket'
>/dev/null || true


/usr/bin/dh_installsystemd
R_FLAG => no restart
RESTART_AFTER_UPGRADE => restart (default)

R_FLAG is only considered in postrm to stop/notstop it
RESTART_AFTER_UPGRADE is considered for postinst

We'd need to set RESTART_AFTER_UPGRADE=0 as well.
That is not (no more?) implied by --no-stop-on-upgrade

First I split list in services and sockets and added the extra arg
just to those not intended to restart:
dh_installsystemd -p libvirt-daemon-system --no-stop-on-upgrade
--no-restart-after-upgrade $(LIBVIRT_SYSTEM_SERVICES_NR)

New section is:
if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" =
"abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then
if [ -d /run/systemd/system ]; then
systemctl --system daemon-reload >/dev/null || true
deb-systemd-invoke start 'virtlockd.service'
'virtlockd.socket' 'virtlogd.service' 'virtlogd.socket' >/dev/null ||
true
fi
fi

And one would think that this would keep the processes it up and running as-is.
This actually worked, but we are somewhat back at the original issue
that the restarting the sockets restarts the services (just without
sysV this time).

Later on come the services which still have "restart"

Main PID: 28688 (virtlogd)
Main PID: 28687 (virtlockd)
+ deb-systemd-invoke restart libvirt-guests.service
virtlockd-admin.socket virtlockd.socket virtlogd-admin.socket
virtlogd.socket
++ grep 'Main PID'
++ systemctl status virtlogd.service virtlockd.service --no-pager --lines 1
Main PID: 29470 (virtlogd)
Main PID: 29469 (virtlockd)

But there isn't really a reason to restart the sockets at all.
And the services already have their systemctl reload virtlogd.service
section in postinst for the proper re-exec.
So lets just make the sockets --no-stop-on-upgrade +
--no-restart-after-upgrade as well.

This seems to do the trick to achieve the correct behavior.
diff --git a/debian/rules b/debian/rules
index 26fc3e7171..63b8a2a316 100755
--- a/debian/rules
+++ b/debian/rules
@@ -247,7 +247,7 @@ override_dh_installinit:

override_dh_installsystemd:
dh_installsystemd -p libvirt-daemon-system
--restart-after-upgrade libvirtd.service
- dh_installsystemd -p libvirt-daemon-system
--no-stop-on-upgrade $(LIBVIRT_SYSTEM_SERVICES)
+ dh_installsystemd -p libvirt-daemon-system
--no-stop-on-upgrade --no-restart-after-upgrade
$(LIBVIRT_SYSTEM_SERVICES)

override_dh_installdocs:
dh_installdocs -plibvirt-doc --doc-main-package libvirt-doc

I have not yet tried what happens if I let the sysV scripts back in.
But for systemd only this seems worth to discuss.

[Christian Ehrhardt]

Testing now confirmed, that for the version in experimental I need to do both:

a) drop the sysV
- as Ubuntu has done for a while
- without the sysV to systemd mapping still restarts the services
- something like [1], I haven an MP up for that on salsa, might be
slightly outdated

b) Specify both --no-stop-on-upgrade and --no-restart-after-upgrade
Otherwise dh_installsystemd snippets will restart the service

[1]: https://git.launchpad.net/~libvirt-maintainers/ubuntu/+source/libvirt/commit/?id=0b5b15b390903e5c282a8bb2c27d53d63e442f31

[Guido Günther]

Hi,
Thanks for a lot for testing this!

On Tue, Jun 11, 2019 at 09:05:43PM +0200, Christian Ehrhardt wrote:
> Testing now confirmed, that for the version in experimental I need to do both:
>
> a) drop the sysV
> - as Ubuntu has done for a while
> - without the sysV to systemd mapping still restarts the services
> - something like [1], I haven an MP up for that on salsa, might be
> slightly outdated
>
> b) Specify both --no-stop-on-upgrade and --no-restart-after-upgrade
> Otherwise dh_installsystemd snippets will restart the service

That would mean three things for buster:

a) switch to dh12 (since we have that for newer versions since a while
the change wouldn't be too bad)
b) add --no-stop-on-upgrade
c) Drop the sysv init scripts

While we could do the first two c) wouldn't be nice and we'd need to add
an explicit dependency on systemd-sysv. I'll try to get a round to test
this next week.

Cheers,
-- Guido

[Karl Goetz]

On Tue, 11 Jun 2019 21:05:43 +0200 Christian Ehrhardt
<christian...@canonical.com> wrote:
> Testing now confirmed, that for the version in experimental I need to do both:
>
> a) drop the sysV
> - as Ubuntu has done for a while
> - without the sysV to systemd mapping still restarts the services
> - something like [1], I haven an MP up for that on salsa, might be
> slightly outdated
>
> b) Specify both --no-stop-on-upgrade and --no-restart-after-upgrade
> Otherwise dh_installsystemd snippets will restart the service
>
> [1]: https://git.launchpad.net/~libvirt-maintainers/ubuntu/+source/libvirt/commit/?id=0b5b15b390903e5c282a8bb2c27d53d63e442f31
>

Hi,
A quick word of thanks - removing the sysv script made the upgrade from
stretch to buster complete for me.

In case you notice its missing, I had previously commented out
'virtlogd-admin.socket' from line 386 of the postinst. I'm unsure if it
was related to having the install complete but will test another time.

thanks,
Karl.

> root@mainserver:/etc# apt dist-upgrade

> Reading package lists... Done
> Building dependency tree
> Reading state information... Done

> Calculating upgrade... Done

> The following packages were automatically installed and are no longer required:

> adwaita-icon-theme arch-test at-spi2-core busybox-static cloud-image-utils dconf-gsettings-backend dconf-service dh-python distro-info e2fsprogs-l10n ebtables fontconfig freeipmi-common glib-networking glib-networking-common
> glib-networking-services gnupg-agent gsettings-desktop-schemas gstreamer1.0-libav gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-ugly gstreamer1.0-x gtk-update-icon-cache guile-2.0-libs hicolor-icon-theme
> i965-va-driver ibverbs-providers intel-media-va-driver irqbalance liba52-0.7.4 libaa1 libaacs0 libaom0 libass9 libasyncns0 libatk-bridge2.0-0 libatk1.0-0 libatk1.0-data libatspi2.0-0 libavc1394-0 libavcodec58 libavfilter7
> libavformat58 libavutil56 libbdplus0 libblockdev-crypto2 libbluray2 libboost-atomic1.67.0 libboost-filesystem1.62.0 libboost-iostreams1.62.0 libboost-random1.62.0 libboost-regex1.67.0 libboost-system1.62.0 libboost-thread1.62.0
> libboost-thread1.67.0 libbs2b0 libcaca0 libcdio18 libcdparanoia0 libchromaprint1 libcodec2-0.8.1 libcolord2 libcroco3 libcrystalhd3 libcups2 libdatrie1 libdconf1 libdrm-amdgpu1 libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libdv4
> libdvdnav4 libdvdread4 libevent-2.0-5 libevtlog0 libfftw3-double3 libflac8 libflite1 libfreeipmi17 libgdbm3 libgdk-pixbuf2.0-0 libgdk-pixbuf2.0-bin libgdk-pixbuf2.0-common libgl1 libgl1-mesa-dri libglapi-mesa libglvnd0 libglx-mesa0
> libglx0 libgme0 libgomp1 libgpg-error-l10n libgpgme11 libgraphite2-3 libgsm1 libgtk-3-0 libgtk-3-bin libgtk-3-common libharfbuzz0b libice6 libicu57 libiec61883-0 libigdgmm5 libjack-jackd2-0 libjansson4 libjbig0 libjson-glib-1.0-0
> libjson-glib-1.0-common liblcms2-2 liblilv-0-0 libllvm7 liblvm2app2.2 liblvm2cmd2.02 libminiupnpc10 libmp3lame0 libmpeg2-4 libmpg123-0 libmysofa0 libnftables0 libnorm1 libnss-systemd libntfs-3g871 libogg0 libopencore-amrnb0
> libopencore-amrwb0 libopenipmi0 libopenjp2-7 libopenmpt0 libpam-cap libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0 libperl5.24 libpgm-5.2-0 libpostproc55 libproxy1v5 libpulse0 libpython3.5-minimal libpython3.5-stdlib librados2
> libraw1394-11 librbd1 librest-0.7-0 librsvg2-2 librsvg2-common librubberband2 libsamplerate0 libsdl1.2debian libsensors-config libsensors5 libserd-0-0 libshine3 libshout3 libsidplay1v5 libslang2 libsm6 libsnappy1v5 libsndfile1
> libsnmp-base libsnmp30 libsodium23 libsord-0-0 libsoup-gnome2.4-1 libsoup2.4-1 libsoxr0 libspeex1 libsratom-0-0 libssh-gcrypt-4 libswresample3 libswscale5 libtag1v5 libtag1v5-vanilla libthai-data libthai0 libtheora0 libtiff5
> libtwolame0 libunistring0 libv4l-0 libv4lconvert0 libva-drm2 libva-x11-2 libva2 libvdpau-va-gl1 libvdpau1 libvidstab1.1 libvisual-0.4-0 libvolume-key1 libvorbis0a libvorbisenc2 libvorbisfile3 libvpx5 libvte-2.91-0 libvte-2.91-common
> libwavpack1 libwayland-client0 libwayland-cursor0 libwayland-egl1 libwebp6 libwebpmux3 libx11-xcb1 libx264-155 libx265-165 libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-sync1 libxcb-xfixes0 libxcomposite1 libxcursor1
> libxdamage1 libxen-4.8 libxfixes3 libxi6 libxinerama1 libxkbcommon0 libxml2-utils libxrandr2 libxshmfence1 libxtst6 libxv1 libxvidcore4 libxxf86vm1 libzmq5 libzvbi-common libzvbi0 linux-image-4.9.0-7-amd64 lxc-templates
> mesa-va-drivers mesa-vdpau-drivers netcat-openbsd nftables openipmi ovmf pigz python-certifi python-chardet python-gi python-idna python-ipaddr python-libvirt python-libxml2 python-pkg-resources python-requests python-six
> python-urllib3 python3.5 python3.5-minimal qemu-system-gui runit-helper sgml-base shared-mime-info thin-provisioning-tools transmission-cli uuid-runtime va-driver-all vdpau-driver-all x11-common xkb-data xml-core

> Use 'apt autoremove' to remove them.

> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
> 1 not fully installed or removed.
> After this operation, 0 B of additional disk space will be used.

> Do you want to continue? [Y/n]

> Setting up libvirt-daemon-system (5.0.0-4) ...
> + [ configure = configure ]
> + [ -d /run/systemd/system ]
> + systemctl --system daemon-reload
> + [ -n 3.0.0-4+deb9u4 ]
> + _dh_action=restart
> + deb-systemd-invoke restart virtlockd.socket
> + [ configure = configure ]
> + [ -d /run/systemd/system ]
> + systemctl --system daemon-reload
> + deb-systemd-invoke start virtlockd-admin.socket virtlockd.service virtlockd.socket

> virtlockd.service is a disabled or a static unit, not starting it.

> + dpkg-maintscript-helper rm_conffile /etc/logrotate.d/libvirtd.uml 5.0.0-2~ -- configure 3.0.0-4+deb9u4
> + [ configure = configure ]
> + [ -x /etc/init.d/virtlogd ]
> + update-rc.d virtlogd defaults
> + invoke-rc.d virtlogd start

> A dependency job for virtlogd.service failed. See 'journalctl -xe' for details.
> invoke-rc.d: initscript virtlogd, action "start" failed.
> ● virtlogd.service - Virtual machine log manager
> Loaded: loaded (/lib/systemd/system/virtlogd.service; indirect; vendor preset: enabled)

> Active: active (running) since Fri 2019-08-30 10:26:16 UTC; 1h 28min ago

> Docs: man:virtlogd(8)
> https://libvirt.org

> Main PID: 5885 (virtlogd)
> Tasks: 2 (limit: 4915)
> Memory: 5.6M
> CGroup: /system.slice/virtlogd.service
> └─5885 /usr/sbin/virtlogd
>
> Aug 30 11:51:08 mainserver systemd[1]: Dependency failed for Virtual machine log manager.
> Aug 30 11:51:08 mainserver systemd[1]: virtlogd.service: Job virtlogd.service/start failed with result 'dependency'.
> Aug 30 11:53:14 mainserver systemd[1]: Reloading Virtual machine log manager.
> Aug 30 11:53:14 mainserver systemd[1]: Reloaded Virtual machine log manager.
> Aug 30 11:53:19 mainserver systemd[1]: Dependency failed for Virtual machine log manager.
> Aug 30 11:53:19 mainserver systemd[1]: virtlogd.service: Job virtlogd.service/start failed with result 'dependency'.
> Aug 30 11:54:48 mainserver systemd[1]: Reloading Virtual machine log manager.
> Aug 30 11:54:48 mainserver systemd[1]: Reloaded Virtual machine log manager.
> Aug 30 11:54:53 mainserver systemd[1]: Dependency failed for Virtual machine log manager.
> Aug 30 11:54:53 mainserver systemd[1]: virtlogd.service: Job virtlogd.service/start failed with result 'dependency'.
> + exit 1

> dpkg: error processing package libvirt-daemon-system (--configure):

> installed libvirt-daemon-system package post-installation script subprocess returned error exit status 1

> Errors were encountered while processing:
> libvirt-daemon-system
> E: Sub-process /usr/bin/dpkg returned an error code (1)

> root@mainserver:/etc# ls -lh /etc/init.d/virtlogd
> -rwxr-xr-x 1 root root 4.0K Mar 12 2018 /etc/init.d/virtlogd
> root@mainserver:/etc# rm /etc/init.d/virtlogd
> root@mainserver:/etc# apt dist-upgrade

> Reading package lists... Done
> Building dependency tree
> Reading state information... Done

> Calculating upgrade... Done

> The following packages were automatically installed and are no longer required:

> adwaita-icon-theme arch-test at-spi2-core busybox-static cloud-image-utils dconf-gsettings-backend dconf-service dh-python distro-info e2fsprogs-l10n ebtables fontconfig freeipmi-common glib-networking glib-networking-common
> glib-networking-services gnupg-agent gsettings-desktop-schemas gstreamer1.0-libav gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-ugly gstreamer1.0-x gtk-update-icon-cache guile-2.0-libs hicolor-icon-theme
> i965-va-driver ibverbs-providers intel-media-va-driver irqbalance liba52-0.7.4 libaa1 libaacs0 libaom0 libass9 libasyncns0 libatk-bridge2.0-0 libatk1.0-0 libatk1.0-data libatspi2.0-0 libavc1394-0 libavcodec58 libavfilter7
> libavformat58 libavutil56 libbdplus0 libblockdev-crypto2 libbluray2 libboost-atomic1.67.0 libboost-filesystem1.62.0 libboost-iostreams1.62.0 libboost-random1.62.0 libboost-regex1.67.0 libboost-system1.62.0 libboost-thread1.62.0
> libboost-thread1.67.0 libbs2b0 libcaca0 libcdio18 libcdparanoia0 libchromaprint1 libcodec2-0.8.1 libcolord2 libcroco3 libcrystalhd3 libcups2 libdatrie1 libdconf1 libdrm-amdgpu1 libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libdv4
> libdvdnav4 libdvdread4 libevent-2.0-5 libevtlog0 libfftw3-double3 libflac8 libflite1 libfreeipmi17 libgdbm3 libgdk-pixbuf2.0-0 libgdk-pixbuf2.0-bin libgdk-pixbuf2.0-common libgl1 libgl1-mesa-dri libglapi-mesa libglvnd0 libglx-mesa0
> libglx0 libgme0 libgomp1 libgpg-error-l10n libgpgme11 libgraphite2-3 libgsm1 libgtk-3-0 libgtk-3-bin libgtk-3-common libharfbuzz0b libice6 libicu57 libiec61883-0 libigdgmm5 libjack-jackd2-0 libjansson4 libjbig0 libjson-glib-1.0-0
> libjson-glib-1.0-common liblcms2-2 liblilv-0-0 libllvm7 liblvm2app2.2 liblvm2cmd2.02 libminiupnpc10 libmp3lame0 libmpeg2-4 libmpg123-0 libmysofa0 libnftables0 libnorm1 libnss-systemd libntfs-3g871 libogg0 libopencore-amrnb0
> libopencore-amrwb0 libopenipmi0 libopenjp2-7 libopenmpt0 libpam-cap libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0 libperl5.24 libpgm-5.2-0 libpostproc55 libproxy1v5 libpulse0 libpython3.5-minimal libpython3.5-stdlib librados2
> libraw1394-11 librbd1 librest-0.7-0 librsvg2-2 librsvg2-common librubberband2 libsamplerate0 libsdl1.2debian libsensors-config libsensors5 libserd-0-0 libshine3 libshout3 libsidplay1v5 libslang2 libsm6 libsnappy1v5 libsndfile1
> libsnmp-base libsnmp30 libsodium23 libsord-0-0 libsoup-gnome2.4-1 libsoup2.4-1 libsoxr0 libspeex1 libsratom-0-0 libssh-gcrypt-4 libswresample3 libswscale5 libtag1v5 libtag1v5-vanilla libthai-data libthai0 libtheora0 libtiff5
> libtwolame0 libunistring0 libv4l-0 libv4lconvert0 libva-drm2 libva-x11-2 libva2 libvdpau-va-gl1 libvdpau1 libvidstab1.1 libvisual-0.4-0 libvolume-key1 libvorbis0a libvorbisenc2 libvorbisfile3 libvpx5 libvte-2.91-0 libvte-2.91-common
> libwavpack1 libwayland-client0 libwayland-cursor0 libwayland-egl1 libwebp6 libwebpmux3 libx11-xcb1 libx264-155 libx265-165 libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-sync1 libxcb-xfixes0 libxcomposite1 libxcursor1
> libxdamage1 libxen-4.8 libxfixes3 libxi6 libxinerama1 libxkbcommon0 libxml2-utils libxrandr2 libxshmfence1 libxtst6 libxv1 libxvidcore4 libxxf86vm1 libzmq5 libzvbi-common libzvbi0 linux-image-4.9.0-7-amd64 lxc-templates
> mesa-va-drivers mesa-vdpau-drivers netcat-openbsd nftables openipmi ovmf pigz python-certifi python-chardet python-gi python-idna python-ipaddr python-libvirt python-libxml2 python-pkg-resources python-requests python-six
> python-urllib3 python3.5 python3.5-minimal qemu-system-gui runit-helper sgml-base shared-mime-info thin-provisioning-tools transmission-cli uuid-runtime va-driver-all vdpau-driver-all x11-common xkb-data xml-core

> Use 'apt autoremove' to remove them.

> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
> 1 not fully installed or removed.
> After this operation, 0 B of additional disk space will be used.

> Do you want to continue? [Y/n]

> [master b0be562] saving uncommitted changes in /etc prior to apt run
> 2 files changed, 162 deletions(-)
> delete mode 100755 init.d/virtlogd
> Setting up libvirt-daemon-system (5.0.0-4) ...
> + [ configure = configure ]
> + [ -d /run/systemd/system ]
> + systemctl --system daemon-reload
> + [ -n 3.0.0-4+deb9u4 ]
> + _dh_action=restart
> + deb-systemd-invoke restart virtlockd.socket
> + [ configure = configure ]
> + [ -d /run/systemd/system ]
> + systemctl --system daemon-reload
> + deb-systemd-invoke start virtlockd-admin.socket virtlockd.service virtlockd.socket

> virtlockd.service is a disabled or a static unit, not starting it.

> + dpkg-maintscript-helper rm_conffile /etc/logrotate.d/libvirtd.uml 5.0.0-2~ -- configure 3.0.0-4+deb9u4
> + [ configure = configure ]
> + [ -x /etc/init.d/virtlogd ]
> + [ configure = configure ]
> + [ -x /etc/init.d/libvirtd ]
> + update-rc.d libvirtd defaults 28 72
> + [ -n 3.0.0-4+deb9u4 ]
> + _dh_action=restart
> + invoke-rc.d libvirtd restart
> + [ configure = configure ]
> + [ -x /etc/init.d/libvirt-guests ]
> + update-rc.d libvirt-guests defaults 29 71
> + invoke-rc.d libvirt-guests start
> + exit 0
> root@mainserver:/etc#