4.0-rc3: sys-net not getting updated template OS image?

[Steve Coleman]

I have a strange situation where my sys-net's software template
"fedora-26-net" (variant of fedora-minimal) does not appear to be
providing updated OS images. My sys-net is the only vm using this
specific image.

I have a necessary network service installed in the template
"fedora-26-net" /opt/service-name directory, installed from an rpm,
which was working perfectly until I was forced to recover from backup. I
have been trying to get it working as it once was to no avail. Every
time I boot up the networking vm's I now get a sys-net without my
required /opt/service-name directory installed. The template has it, and
sys-net uses that template, but sys-net never seems to get a new copy of
it.

The /opt/service-name directory in sys-net is not even there, and one
sub-directory that is not even in that template anymore *is* there, and
it even starts up that service which I do not want running in sys-net.

Each time I want to connect to the corporate network I now have to kill
one service, reinstall that required service directly in sys-net COW,
just to temporarily create that required service so I can connect. That
modification of course goes away just as soon as sys-net is shut down,
so this gets repeated often.

As another test, I created a brand new user vm (test-fedora-26-net)
based on fedora-26-net and opened a terminal there, and one in the
template itself. The /opt directories were different. The
test-fedora-26-net has a file structure that should never have existed
when it was created.

Any idea how to force this rogue and defiant template to pass along a
new OS image? Apparently doing a "dnf update" in the template itself
isn't enough to kick it into gear and get it to happen. Changing the
source template, and changing it back again does not trigger it either.
Even creating a new vm gets an old copy!

I think I may need to buy a clue. Is this somehow a qvm-feature related
thing? A qubes-rpc not happening? What signaling is supposed to happen
that isn't?

thanks,

Steve.

[awokd]

On Tue, February 13, 2018 6:32 pm, Steve Coleman wrote:
> I have a strange situation where my sys-net's software template
> "fedora-26-net" (variant of fedora-minimal) does not appear to be
> providing updated OS images. My sys-net is the only vm using this specific
> image.

Are you running low on disk space?

Not exactly sure what's going on there, but can you try to clone the
problem template and point your AppVM to the clone instead, and see if
that works?

[Steve Coleman]

This template is only 30% used.

As was suggested, I cloned the template to see what happened. There were
no apparent errors when cloning.

- Before testing the template clone with an appvm I did a "dnf update"
in each template (orig & clone). The clone got 135 updates, and the
original said it was already up to date. Clearly cloning is not an exact
copy mechanism.

- To check why the difference in the updates, an 'rpm -qa' verified that
they are both now on the exact same version #'s of the updated packages
that I checked, so the original template was not lying to me that it was
"up to date". How the copy could be out of date is then quite puzzling.

- As for the /opt directory found in each, the clone has the invalid
subdirectories in /opt, and the original template has the correct set of
directories. So cloning works just the same as OS provisioning in that
it collects the wrong data, and copies that.

- As for the planned appvm test, when using the cloned template, it
obviously saw the wrong /opt directories which were in the cloned template.

My plan to move myself forward is to update the new clone with the right
software packages and swap that template into sys-net, removing the bad
template.

But, understanding how this kind of corruption happened could be
important. Something is obviously broken with this original template, so
I went to look for any sign of cross-linked inodes in the image files or
directory structures, only to find that Qubes 4.0 totally changed the
way that VM image files are stored and handled. Wow, I don't even know
where to begin. An fsck in dom0 says everything is fine there. I have no
clue as to what constitutes an appvm filesystem image in the new design.



Steve

[Rusty Bird]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Steve Coleman:

> I have a strange situation where my sys-net's software template
> "fedora-26-net" (variant of fedora-minimal) does not appear to be providing
> updated OS images. My sys-net is the only vm using this specific image.

Assuming that sys-net is _not_ a DispVM, maybe this is still somehow
similar to https://github.com/QubesOS/qubes-issues/issues/3576 - can
you search for '<property name="name">sys-net</property>' in dom0's
/var/lib/qubes/qubes.xml and post the next (i.e. somewhere below that
line) XML '<volume-config>' block?

Rusty
-----BEGIN PGP SIGNATURE-----

iQJ8BAEBCgBmBQJahG6JXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4NEI1OUJDRkM2MkIxMjlGRTFCMDZEMDQ0
NjlENzhGNDdBQUYyQURGAAoJEEadePR6ryrfmwsQAIw0uxVBLbrz543iD0z6EqBn
doi1XooXsmoqfaFXvSqnJSJlVr1vhgFM+mYVzpPIOvT20AhzgYDIP1YWjU/KWO3q
DNKkpLuzJqhPfG2XTv63MGmji/Tz1vLMK6OGTDSU7THR8DgKJSfy/3bBRJP9meeq
x4BzUmTG7HVUsjbRO7AK1QO4XaCwi8sFCsnRNht4UsL0ulSeAG1gaVxR9zxSul2U
hqLW8SGk+zgMbsdY8pNjreUNmMK63r9auIAiIapoU9s13g42P7E3ujI9HX2iO8C0
PXQfmNo2U1frSycySLn5kn8Qa94uLvLTf6Xj2985EPsmicYH85PdGKwVRsa0ObLQ
/RrWrbxbximu8g8onbPnqMig5Nu56yyMD/m4CY7PDaqh/IN1cK3e9tBbO574QuHH
kkVgmzC9ojrIXDk7qH2Tp5M2P6KtSNHHQ2NXiImuxDxW6ZV47AnjIngk5Rw4bfCI
xrv1EU/Ujxa0ioosm4NkC6tPb2bqbRoHYqiEZigx4/onpV1YM+mgsHNLoopFJ3qY
QqMu9YFTJv9IY2/f6NsVLuWh0QU1csBXETr41UOZwB/AnOqolm0jrZADD+8EjnQ6
My+WUCm+WZWKz55iuFPVDxscRSWxQeMOc7msyKACk0Ej6Th1RJcc1yKt20eIw9QA
XA0EWlMJksbqJaqyVgmO
=UNib
-----END PGP SIGNATURE-----

[Steve Coleman]

On 02/14/18 12:14, Rusty Bird wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> Steve Coleman:
>> I have a strange situation where my sys-net's software template
>> "fedora-26-net" (variant of fedora-minimal) does not appear to be providing
>> updated OS images. My sys-net is the only vm using this specific image.
>
> Assuming that sys-net is _not_ a DispVM, maybe this is still somehow
> similar to https://github.com/QubesOS/qubes-issues/issues/3576 - can
> you search for '<property name="name">sys-net</property>' in dom0's
> /var/lib/qubes/qubes.xml and post the next (i.e. somewhere below that
> line) XML '<volume-config>' block?
>

I looked at the github issue and I'm not certain of any similarity. I
did copy another template vm to make sys-net's template, but I can not
remember copying or renaming the sys-net vm itself.

Here is the sys-net <volume-config>. I re-wraped the xml to make it a
little more readable in email:

<volume-config>
<volume name="root"
pool="lvm"
revisions_to_keep="0"
size="21474836480"
snap_on_start="True"
source="qubes_dom0/vm-fedora-26-net-root"
vid="qubes_dom0/vm-sys-net-root"/>
<volume name="private"
pool="lvm"
revisions_to_keep="0"
rw="True"
save_on_stop="True"
size="2147483648"
vid="qubes_dom0/vm-sys-net-private"/>
<volume name="kernel"
pool="linux-kernel"
revisions_to_keep="0"
vid="4.14.13-3"/>
<volume name="volatile"
pool="lvm"
revisions_to_keep="0"
rw="True"
size="10737418240"
vid="qubes_dom0/vm-sys-net-volatile"/>
</volume-config>


Since it is the template (fedora-26-net) itself that appears to be
broken, would that not be what needs to be verified? Here is the
fedora-26-net template config:

<volume-config>
<volume name="root"
pool="lvm"
revisions_to_keep="0"
rw="True"
save_on_stop="True"
size="21474836480"
vid="qubes_dom0/vm-fedora-26-net-root"/>
<volume name="kernel"
pool="linux-kernel"
revisions_to_keep="0"
vid="4.14.13-3"/>
<volume name="private"
pool="lvm"
revisions_to_keep="0"
rw="True"
save_on_stop="True"
size="4294967296"
vid="qubes_dom0/vm-fedora-26-net-private"/>
<volume name="volatile"
pool="lvm"
revisions_to_keep="0"
rw="True"
size="10737418240"
vid="qubes_dom0/vm-fedora-26-net-volatile"/>
</volume-config>

if there is a problem there I'm just not seeing it.

thanks,

Steve

[Rusty Bird]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Steve Coleman:

> Here is the sys-net <volume-config>. I re-wraped the xml to make it a little
> more readable in email:
>
> <volume-config>
> <volume name="root"
> pool="lvm"
> revisions_to_keep="0"
> size="21474836480"
> snap_on_start="True"
> source="qubes_dom0/vm-fedora-26-net-root"
> vid="qubes_dom0/vm-sys-net-root"/>

Looks good.

> Since it is the template (fedora-26-net) itself that appears to be broken,
> would that not be what needs to be verified?

I had a hunch that somehow the wrong template might have ended up as
the source volume for sys-net's root volume, but apparently not.

Rusty
-----BEGIN PGP SIGNATURE-----

iQJ8BAEBCgBmBQJahMpEXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4NEI1OUJDRkM2MkIxMjlGRTFCMDZEMDQ0
NjlENzhGNDdBQUYyQURGAAoJEEadePR6ryrf408P/1JnKeJDyxbbnFQZHLdphFod
ZvR6G3IAzjKm1ekqfXiCedbRXFCK35Olw+PO+r0MEVMvHH414CvcuN9Fgf0LyqHC
qtGHnPl1eaxt/i7Wt8h/GKd+9bxysE9CtV43JntSAYd4atbTiv7FMSUOPgZyanTo
ycEAPdL96LtTYUtD3jR/FOv+1OHk92mYgQrIWe/LfljEWlrGXj/+yDIHxEpZpAmW
0d600FfqRTGh/QDmb2IcZkrBCc6bR5qh7tyxwA7eq+TywMWdeyCeL7zNpmYNqRHu
tjLgUdQsCo/2OnqF5t1/HDLcTcnen1Yw7sQpl4hdb7E+tnqHyZ04NgMSuM9bERT7
MC2fPxh5d55yY44jLJpwr2pH5W0/Oj/jswHfAtJrf+uNnOJqApJmFKXd6MjU0vzi
JKDbj9L2itt6Q9uni3C7wMKPtLqqq0K/0XIflbBoRa39apHgn7QaHFG+kqmY7Its
rwF3voMUbwGohjwrEGZIh9VG7dMyqK2vAVW0KwG01fpJO7mVsNZoClAxlK6cR6C7
jwEKJszz6vQ1Xfi85o+5OJlk4EV1pUrCSzd0lqdZUQDrZxzZ8EqeRLSlfmGZ8UpL
jP1Dyc1QyY5IdMObmnnAAvf+0ehB2EsFo7jgn0pXPPiiH/7Qf8L5xB2yyEFNKW+z
70Xb+0LZHCscftb46Zsq
=wqE/
-----END PGP SIGNATURE-----