Any chance the freezing could be resolved?

111 views
Skip to first unread message

Drew White

unread,
Nov 20, 2016, 8:03:39 PM11/20/16
to qubes-users
Still getting Dom0 crash / parts not responding.

Primarily it's a guest that causes the whole of Dom0 to slow and stop.

I have yet to find out what the root cause it, but it's still locking things up.

Sometimes after running a guest for a few hours will cause the system to start having a coronary.

I'm not doing anything super intensive, only programming.


IF I run ANY guest with Firefox, and have it running for a couple of days, I come back from the weekend or sometimes even 1 night, and the PC is locked up solid. At that point, not even the logging in Dom0 is working.
Sometimes, as I have said before, the logging in Dom0 is still running and working but I can't get access to the machine any more and have to physically power it down to then start it up again. Which causes an fsck to be run because the filesystem wasn't cleanly unmounted.

Any help on these bugs would be greatly appreciated.
Mainly I find the issue is with FireFox running. I've found other guests running for days on end don't cause the system to lock up.

If information for the developers is required, I would be happy to email you the details, logs, and specs.

Jean-Philippe Ouellet

unread,
Nov 20, 2016, 8:11:39 PM11/20/16
to Drew White, qubes-users
If I were you I would try to see if you can reproduce the issue with
upstream xen, and then ask on the xen mailing list.

It sounds more like a this-xen-version + this-linux-version on
your-hardware problem than a qubes problem.

Drew White

unread,
Nov 20, 2016, 8:44:07 PM11/20/16
to qubes-users, drew....@gmail.com

How do I reproduce the issue on upstream XEN when I run Qubes and keep working and doing my stuff without wasting several weeks on testing it on upstream XEN?

Qubes is downstream XEN I assume from what you are saying, which means that the version of XEN that Qubes uses is modified in some ways?
Which means that it's a different version of XEN altogether?

Jean-Philippe Ouellet

unread,
Nov 20, 2016, 9:40:02 PM11/20/16
to Drew White, qubes-users
On Sun, Nov 20, 2016 at 8:44 PM, Drew White <drew....@gmail.com> wrote:
> How do I reproduce the issue on upstream XEN when I run Qubes and keep working and doing my stuff without wasting several weeks on testing it on upstream XEN?

I don't know, but seeing as you're the only person who reports
experiencing this issue, nobody else can test things to try to narrow
it down for you.

> Qubes is downstream XEN I assume from what you are saying, which means that the version of XEN that Qubes uses is modified in some ways?
> Which means that it's a different version of XEN altogether?

Well, Qubes uses a slightly patched Xen -- the applied patches can be
found here: https://github.com/QubesOS/qubes-vmm-xen

My suggestions for you at this point:

1) Post your output of:
$ cat /etc/qubes-release
$ xl dmesg | head -1
to this list

2) See if there's any way you can get a serial console on your
machine. Either via Intel AMT (definitely easiest if supported) or
perhaps via an internal serial header someplace (if you have the
necessary hardware and technical inclination) to see if xen produces
any useful log output while it is actually hanging. Make sure your xen
loglvl=all while doing so (which should be already set by the qubes
installer).

3) Provide a full detailed description of the behavior you exhibit to
the xen-users list. (And the first thing they will probably ask is if
you can still reproduce the issue with the latest upstream un-patched
Xen...)

Andrew

unread,
Nov 20, 2016, 10:31:10 PM11/20/16
to qubes...@googlegroups.com
Drew White:
Drew,

Is there any chance you are using a recent (Braswell or maybe Broadwell
or similar) low-end Intel CPU? If so, your problem may actually be due
to a bug in the Linux kernel.

For example, I recently acquired just such a system and experienced
seemingly random total system freezes with Qubes 3.2 ranging anywhere
from 5m to 12h after boot. What fixed the problem, or worked around it,
was:
-use the 4.8.* kernel in the unstable repository
-change the "i915.preliminary_hw_support=1" in the kernel boot command
line to "intel_idle.max_cstate=1", which limits the CPU to drawing the
maximum power :-\
-(may be unnecessary) disabling DRI in my dom0 Xorg.conf (I had to
create a new file):
-"NoAccel" "true"
-"DRI" "false"

Let me know if you're affected by the same bug.

Cheers,
Andrew

Drew White

unread,
Nov 20, 2016, 10:58:45 PM11/20/16
to qubes-users, drew....@gmail.com
(will post a copy of this on the xen users forum as well as here on the qubes users forum.)

On Monday, 21 November 2016 13:40:02 UTC+11, Jean-Philippe Ouellet wrote:
> On Sun, Nov 20, 2016 at 8:44 PM, Drew White <drew....@gmail.com> wrote:
> > How do I reproduce the issue on upstream XEN when I run Qubes and keep working and doing my stuff without wasting several weeks on testing it on upstream XEN?
>
> I don't know, but seeing as you're the only person who reports
> experiencing this issue, nobody else can test things to try to narrow
> it down for you.
>
> > Qubes is downstream XEN I assume from what you are saying, which means that the version of XEN that Qubes uses is modified in some ways?
> > Which means that it's a different version of XEN altogether?
>
> Well, Qubes uses a slightly patched Xen -- the applied patches can be
> found here: https://github.com/QubesOS/qubes-vmm-xen

I'll have to take a look at that again, I have the code here.

> My suggestions for you at this point:
>
> 1) Post your output of:
> $ cat /etc/qubes-release
> $ xl dmesg | head -1
> to this list

[{user}@dom0 {folder}]$ cat /etc/qubes-release
Qubes release 3.2 (R3.2)
[{user}@dom0 {folder}]$ xl dmesg | head -1
Xen 4.6.1-20.fc23
[{user}@dom0 {folder}]$

> 2) See if there's any way you can get a serial console on your
> machine. Either via Intel AMT (definitely easiest if supported) or
> perhaps via an internal serial header someplace (if you have the
> necessary hardware and technical inclination) to see if xen produces
> any useful log output while it is actually hanging. Make sure your xen
> loglvl=all while doing so (which should be already set by the qubes
> installer).

Nope, no possibility.
When the PC freezes completely EVERYTHING stops, the whole thing is no longer responding and no processes function.
When the system background logging continues (crond) which I set up to log things every 5 minutes on every virtual as well as dom0.


> 3) Provide a full detailed description of the behavior you exhibit to
> the xen-users list. (And the first thing they will probably ask is if
> you can still reproduce the issue with the latest upstream un-patched
> Xen...)

I leave system on for a couple of days, sometimes even a day, or over night, or sometimes even while I'm using it...

It locks up.

Sometimes logging continues, sometimes it doesn't. (my crond logging script)

----------------

I only have 50 guests.
I only have maybe 10-15 running at any one time.

Normally they use 2048 GB RAM each, not balanced.
Some I have set to 1024 GB RAM, such as NetVM or ProxyVM or other Guests that do not do much other than file sharing, or inter-vm operations or just a firewall. (or dos / my own / others)

Drew White

unread,
Nov 20, 2016, 11:00:23 PM11/20/16
to qubes-users, kyb...@riseup.net


I honestly don't know.

Perhaps you can see the info and let me know if it's similar to yours?

dmi4.txt
pstree.txt
pstreefull.txt
qubes-release.txt
top.txt
uname.txt
xl_dmesg_head-1.txt

Drew White

unread,
Nov 20, 2016, 11:03:12 PM11/20/16
to qubes-users, kyb...@riseup.net
On Monday, 21 November 2016 14:31:10 UTC+11, Andrew wrote:

I have no second CPU in the system at the moment. Not the extra RAM. That' shwy it says there is a second, but there is none.


I have no Xorg.conf anywhere on my system to alter or move and create a new.


So you did EVERYTHING there at once?
Or did you do one thing, check it, then try a different thing, check it, then when they didn't work on their own you tried them in combination?

Marek Marczykowski-Górecki

unread,
Nov 21, 2016, 8:11:17 AM11/21/16
to Drew White, qubes-users
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Sun, Nov 20, 2016 at 07:58:45PM -0800, Drew White wrote:
> [{user}@dom0 {folder}]$ xl dmesg | head -1
> Xen 4.6.1-20.fc23

Have you tried 4.6.3-21 which is already in stable repository?

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJYMvJwAAoJENuP0xzK19cspS0H/3kYr8MkfgqUBZwrWdZu9G0J
EeSW/eUALJ35yjEw16lF+21eadDsGSmNW762exgGoZ5SvlGdl/MgFulwlgWJLcX3
3OlRt4Tq0rbannpjzPmAeOsKkkh7ctFgh8ZOB0KBWqTZiTByD1uCGikjPbdkOlcX
SU6i9UvPapkU1sSFQXhMJRSR81I5FM2ichsjuQaMEPrCFMIFw97pxgY+wLhoaTVt
IHof1TVFUJjUxfGdj8uVXTN2l5Ihb5OGLKd08QnINoAjQPlpePf7r3I7brek6vJ3
wcySGMFzjUurG00k3RR909sQZ7fFvUZ9Nrywj1BpCGyE2f8YGZVPyKFW/XuTSqs=
=JMyu
-----END PGP SIGNATURE-----

Drew White

unread,
Nov 21, 2016, 6:26:33 PM11/21/16
to qubes-users, drew....@gmail.com
Is that still FC23?

Drew White

unread,
Nov 21, 2016, 6:27:44 PM11/21/16
to qubes-users, drew....@gmail.com
On Tuesday, 22 November 2016 00:11:17 UTC+11, Marek Marczykowski-Górecki wrote:
Will that affect all the VMs?
Or is it just for Dom0?

Drew White

unread,
Nov 21, 2016, 6:28:56 PM11/21/16
to qubes-users, drew....@gmail.com
On Tuesday, 22 November 2016 00:11:17 UTC+11, Marek Marczykowski-Górecki wrote:
What is the package name I'd have to install to get that?

Drew White

unread,
Nov 21, 2016, 6:36:06 PM11/21/16
to qubes-users, drew....@gmail.com
did not work...
$ qubes-dom0-update xen*


did work...
$ qubes-dom0-update xen* --action=update

Marek Marczykowski-Górecki

unread,
Nov 21, 2016, 7:34:22 PM11/21/16
to Drew White, qubes-users
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Mon, Nov 21, 2016 at 03:36:05PM -0800, Drew White wrote:
> did not work...
> $ qubes-dom0-update xen*

Another quirk of yum->dnf transformation...

> did work...
> $ qubes-dom0-update xen* --action=update

Did it changed anything?

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJYM5KJAAoJENuP0xzK19csV4UH/2bldPEzU8rgqZp4WdpcKz2b
7XnrclfsiCwPHQYfetZHP/oZmjYDm9/wazKeW7UeP6bAvC3TxoB5/2oiGJT7pnL5
AStMt06ETdYeDAXC3EsJTyHthD7RpTEtPzbgZQ/CGNFoLhp9Gympv9tqUl3zsuZv
qaPdL5jZ1cLoisnNYPs6W/LnzuQBcoHIxwDukRzQN42SkMCXH5yj4AdF6TR078fM
6ln3aHJQsV8cXc9relDWJ/uWx1BvYTCdgBnNKdDwJVDcICetYTElCLOaEE6MrDvU
3cQ01XLH6TPnHLBRWSWLA5wub9P4MkgYGBbOAjiFdX9bJqHjPLyCV5LtK0Oz9ko=
=yGcH
-----END PGP SIGNATURE-----

Andrew

unread,
Nov 22, 2016, 1:27:44 AM11/22/16
to qubes...@googlegroups.com
Drew White:
It doesn't look like this is the same bug, but...

> I have no Xorg.conf anywhere on my system to alter or move and create
a new.

Yes, as I said, I had to create one.

> So you did EVERYTHING there at once?
> Or did you do one thing, check it, then try a different thing, check
it, then when they didn't work on their own you tried them in combination?

Originally I tried them piecemeal, but this is the only combination I
have found that works. I have not tried without the Xorg.conf
modifications but with the other two, though.

Andrew

Drew White

unread,
Nov 22, 2016, 6:21:51 PM11/22/16
to qubes-users, drew....@gmail.com
It doesn't matter with yum or dnf.

yum install xen* installs all xen.
dnf install xen* installs all xen.

Both do the same thing, update what is needed to be updated and also install new things that are xen, including dependancies. (or so I have found every time I do a yum or dnf)

Reply all
Reply to author
Forward
0 new messages