network attach problem after the last updates on 3.2

30 views
Skip to first unread message

Zrubi

unread,
Feb 23, 2018, 5:52:42 AM2/23/18
to qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi,

I have a strange problem after updating my system (3.2) from the
testing repository.

Changing the NetVM of a proxyVM, takes longer time than before, but
succeed - at least according to Qubes manager (and qvm tools)
But after the change, no traffic visible on the virtual interfaces, so
no networking after that move.

I need to reboot the proxyVM (and all the connected AppVMs) to restore
networking.

It seems only affecting ProxyVMs - at least my newly started dipVM's
without netVM are not affected, I can change it's netVM without any issu
e.

Any idea what went wrong?

Thanks.
- --
Zrubi
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEw39Thm3rBIO+xeXXGjNaC1SPN2QFAlqP8mgACgkQGjNaC1SP
N2RmtRAApM/dhapfHuJXv7zK1b7NDykOX8iJqxVE3wW3Zw/Rz9K6MlIQ415MAYyb
7ywrjXf5O8BB50lJR86BbM9cNd6Bipdu/X896HT7tbKQOSfbz/CzbwBOqZH02RwV
uq2JUUFVN93+WAgRkW4XgNiKFaPzIDeF9CxfPykwEIWcGHq1nqWqQWdXAIQ/6Cjs
jNxOlIUcuX2ItFfd2H8ym5s3af6IQ7LUUUFZAtNXrvn44nFZO4bUnCPNj/87lptX
ruwZNNv9GpEie2gEkw/JAoaM/yLriw0M4cK+Ljt+bS5VjSk/4xzo54+waKaGVKDl
7Od44wPkjonj28a8LEnMiIQf/goflA8QslejCQhtYoKIQehNTE94iUGlnt++iyl0
J6zSbYLQLwzhBG7kwwaebKFQgX3f2nGIJgcdHZHxoZjICKzjBXZmoDB9fI57NUue
3x2654eX4UTJHfk0uUvpc0HRz8RvyvrW4Xuy1yL7xne7S6agit9V42gu5XtYHg6u
MnYpk9NxFPzRFu/CxyWYlD5EbyHUa3Z48AMqPsCr9wscxWNtvKh7NfS/1WkTX49u
N11pGFqkl4V5lbuM09f3uns3FQJMg2oqyuEyg8yoQj+wis2/kBhU6yvEYnAtJzNi
lTzU93x83SDqyPjs5XY0lmINlKnlFc3H+eJnWaXuoIQRlUVj0aI=
=indN
-----END PGP SIGNATURE-----

Zrubi

unread,
Feb 23, 2018, 6:08:13 AM2/23/18
to qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 02/23/2018 11:52 AM, Zrubi wrote:
> I have a strange problem after updating my system (3.2) from the
> testing repository.
>
> Changing the NetVM of a proxyVM, takes longer time than before,
> but succeed - at least according to Qubes manager (and qvm tools)
> But after the change, no traffic visible on the virtual interfaces,
> so no networking after that move.

More details:

There is no error messages related this issue.
But If I want to detach this bugged ProxyVM from the connected AppVM
(means: If try to change the connected AppVM's netVM) it is failing
with the following error message:

Internal error: libxenlight failed to detach network device

- --
Zrubi
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEw39Thm3rBIO+xeXXGjNaC1SPN2QFAlqP9gwACgkQGjNaC1SP
N2QV1BAAiO5ddDajHGV3Q/Avt+13X3tDnfrkzCfqZd63LF3VVpHqTG82xHQmDESU
x5HHZf4Uaq/Tj9oWMVpFbcrU6RpjDOXvteVnzRwI/i+bAQlk+O4rWNi4vR0+zH6G
rb1RrxDZ3dy18m1exeH72hodOIK2ec/FIEHD33ROYPW4Ey9aVGOpsTyLlT/U1LyA
IB1BSF6/T804eR5xAquADrtPc4PLLdLQbRF6hV5T28dobOcaB+y2rkBURDJywNvB
zS/4GQNY+eZAr/1JAjDD7YidqF+xWWMTJGJoi5+1vrx7RA0EcPJ0GKreLnFXJSGO
7AK1sQfJ1x2iwQly8FjustqCIhALiMAIkzcmhGkCT5dcKWccbYeJbWSnIHLHN6Bn
7nKgsSh5Mz/OwqPBi07Q8Qa+RF9IUSrnQW65solauTMe4u0prAOiKYL/Ey/eOBi0
9klAgIhDPBXicKzONsLiup6Nms7Vl+5CDgylR2wmjuXeURrReh0IihTINP5khenV
+Bgk2wuoTFuwbqFR2sEGL0rBCCRYmBTgvOmH15sRYIIl8GdpeSagcA3wLBxH3Urn
Dt5SzJvzL9DVdsHXVD7VMWQSCzrzYbejDcyTwtgMJaJqsfGxO4oueMS+gFeuK5w2
2b/QVUZag3TCnKX5SB9plFfWMXIk1eCiFcJCqADdpcSTA+G1Q1s=
=OG9u
-----END PGP SIGNATURE-----

M. Vefa Bicakci

unread,
Feb 23, 2018, 8:43:11 PM2/23/18
to Zrubi, qubes...@googlegroups.com
> On 02/23/2018 11:52 AM, Zrubi wrote:
>> I have a strange problem after updating my system (3.2) from the
>> testing repository.
>>
>> Changing the NetVM of a proxyVM, takes longer time than before,
>> but succeed - at least according to Qubes manager (and qvm tools)
>> But after the change, no traffic visible on the virtual interfaces,
>> so no networking after that move.
>
> More details:
>
> There is no error messages related this issue.
> But If I want to detach this bugged ProxyVM from the connected AppVM
> (means: If try to change the connected AppVM's netVM) it is failing
> with the following error message:
>
> Internal error: libxenlight failed to detach network device
>
> - --
> Zrubi

Hello Zrubi,

Are you using Linux kernel version "4.14.18-1.pvops.qubes.x86_64"?

Are you experiencing higher than normal CPU utilization within the
affected VMs due to the "xenbus" and "xenwatch" kernel threads?

If the answers are "yes", I believe you have encountered the same issue
I reported earlier. Here is a link to my report:

https://groups.google.com/d/msg/qubes-devel/ER9v3jy0EeI/4VxLzr4eBQAJ

I believe this is a regression in the kernel. Could you try to revert the
commit mentioned in my report and see if that resolves the issue?

Hope this helps!

Vefa

Zrubi

unread,
Feb 24, 2018, 4:11:01 AM2/24/18
to M. Vefa Bicakci, qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 02/24/2018 02:42 AM, M. Vefa Bicakci wrote:
> Are you using Linux kernel version "4.14.18-1.pvops.qubes.x86_64"?


4.4.62-12 in dom0
4.14.18-1 in VM's

> Are you experiencing higher than normal CPU utilization within the
> affected VMs due to the "xenbus" and "xenwatch" kernel threads?

no, I didn't notice any unusual behavior other than broken networking.


And I can reproduce the issue with 100% "success" rate, by simply
changing netvm. The issue will hit me after the first change.

- --
Zrubi
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEw39Thm3rBIO+xeXXGjNaC1SPN2QFAlqRLBMACgkQGjNaC1SP
N2RBtw/9Gbd4df4lVROcuzd29bARMAmiNM3/UMkBLMf0YzJSCSx8Y0tGiU7xfIbZ
mNRiTULJaL6WQSrzt8BL0lojnD9jpUnJcgEjClXMR2y+kaYLHSNUdsYsjVfWx1U7
g5CYgOMv5a7YPcksiwaXUZIaHJrD6yvE6MMysYH2YnZrh8vQsmq81dmfhxaFW78D
qN8stbkYgx109/joj+MaEwv8Jmj5f20l164dgEmfhfIZp2K8Vk0KPw7Y54xIJpUp
25hs71bTrK9pLH/nbWZg1YtsNDkR/eneYnb2n9LFLw47dgRNhVET6XFleXNs9N+p
hp7tV1N6aFZ085dt0+HeNxMpnowpHkkjIUFDT61u/K7i9wOLHi1ljwhxjlXO9bjQ
ifeAW1uOBEy2NGy02e6UGayn2Wit0i19XrU8pqApiDeDc4hj13G+dlGhABWte/Cg
CQlq9Pdl2RdXAFNhC3yEPFcXw8byhrEWJEHmen6KNnfGGni8pdrUW9/HNS0i6HOm
97xNL06LlRyTPhQIlnSswsCcFcUYZA9MNZoxhgmBIikF2870aPKUlUC18z4J3gS9
tByWr2XCpMFqRU8vAWglUlVB9c6XaSFBfwopwL/kjiF8/d2W7FOhp7OSLF+w2l2m
zC35pwocALyzLpukYwNZ5r6HkTISvRUOnJ5wYeQMLXx84lPyk3U=
=mlz6
-----END PGP SIGNATURE-----

Zrubi

unread,
Feb 28, 2018, 7:59:40 AM2/28/18
to M. Vefa Bicakci, qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 02/24/2018 02:42 AM, M. Vefa Bicakci wrote:
> Are you using Linux kernel version "4.14.18-1.pvops.qubes.x86_64"?
>
> Are you experiencing higher than normal CPU utilization within the
> affected VMs due to the "xenbus" and "xenwatch" kernel threads?

As I had to find the root cause of my network attach problem I tried
many thing including the latest testing kernel: 4.14.18

and I can confirm this is causing a different - but real problem as
you described.

I do not suggest this kernel for daily usage.


- --
Zrubi
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEw39Thm3rBIO+xeXXGjNaC1SPN2QFAlqWp64ACgkQGjNaC1SP
N2RDBA/+NXxeDxQRxlUxY//g11N8hOd6zAyY6PL1kxBnwlklX2kHNnCVKKfP06Cd
+tyJGIHskENz1n0YmUpYvn4b+TC3mu20yixpN7vt/Y0lapOZ0uHe7nUqJGHfuJhX
MQqshVF4hg2P9xV+6ynbfSpl03HC5fA8blB2XIkHObAVJD3rkzkXTqOFw3OLTt+d
gfMnxv+9Xe+g0vO4ovZ7pVGUhpjtT8X1YD/ksUzOXdvlSfrZib+t6EaZ4m0v4cTM
IGvtsHZyrYVzrIE5NVDctIUEN6uCfuyIfLNz0HDQRsrbQgRR3jlcnGX2CnWul/SS
C2106KrGB6319o1yiNTs7FTZyZ9Qd0p7WYQahtnVhSF1RbNDaWDhT3T5QbSqNVTZ
0ucH+sofrQ8TDssDl7lnwcGzgMKiMcfayiPXhJbbIFPAmqBGaD+/lpB59ge3ZsBt
yamp8yvwXKK2zGz5sc45ni7Eo3gIswA5r+VAUoe7mXMXB7+herCs8/1jolszZZif
rW2yobkunfF8V4xZOe/aS+zY4H00nxtIEsmAHlblYJ92z+XrYGKulJ3bbk+4O16U
+FW7Ne07wepHY3Lh+pTh1XpzSY17AjTQfdv65isESaOjKQPNv81PfpGx2hCSE9Nj
Ux1JyslbA4OUycVeMCveFhIfMDf3oYadDpS3sizHKocGBiE2utk=
=h7hG
-----END PGP SIGNATURE-----

Zrubi

unread,
Feb 28, 2018, 8:06:06 AM2/28/18
to qubes...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 02/24/2018 10:10 AM, Zrubi wrote:
> 4.4.62-12 in dom0 4.14.18-1 in VM's

it seems the problem caused by the VM kernel.

Probably it is related to the dom0 problems described in other posts,
and causing my network attaching issues as well.


It seem that in case of using 4.14.12 kernel in dom0 and in VM's
everything is working fine.


ANY newer kernel causing several xen related issues including the
problem I described before.


(while I was downgrading kernels, I got anther issue related to VM's
default kernel. More details coming in a separate thread/issue.)
- --
Zrubi
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEw39Thm3rBIO+xeXXGjNaC1SPN2QFAlqWqS8ACgkQGjNaC1SP
N2QVURAAl1thZo6Kq5ASP92qCW1JGaiO9gq+kFN7TsaLoYPDfxJRj4VsAOPmhijA
tf1MFbWi1Ttrd2CzH9NqVd3CvZV08JZIi29zSattXAb+jf2wxzLMQ71qIg0zYGCv
tquZCuHJrh43MV2aR8clt+1TJfPy05YFCPxwynTGwbxjL0JLThxRzfeGwDJGaR2L
rSoaBAyAcZbH6I4LOhMMusrEk7bc8xg0MolbEaYwkmjlKNv1ICqaQs7Fq6KQTCQM
1c0TVVoM70jJ37YsDhULzcqXJgKrmNodu6juCPal1qZbTbYZTH7jNgZZZBf1gIlP
WkmrCF3zLIzFhykPPirgMjr2U8NbBeqCNIzubxgNwMVaNH9LphVX0p2g1Tu+AcZt
lzEDCjP37A27U045Usmzs2eWKLL+w+/eXUu8eOctjUjSRQl5BXCyWTKhJjKemIQA
j2o+4mEsRi+463V/qbeE1dB+vwaLdQQ433COGA9CfJJUwgSrKZvH1T14p3PWeOin
d4SoaQ0EokzbCNh4rfppOH0InL+yh2ky6bCqBequPk8rVVJFh/0Ytw1jAJKcsp7D
Wh9KIBkEYNpwlRY29/jZ+G/6nCz/tW8ae5fwrcgmSjgf0wFhTjyFFqSMS/g+ADDk
cVXaU83gk9/WBepk9PuSL1ciLtYxCZDYQ+fl/HPPvXGnPNm3TLo=
=mETc
-----END PGP SIGNATURE-----
Reply all
Reply to author
Forward
0 new messages