Thinkpad X1 Carbon Gen6 suspend/resume issues

[tfm...@googlemail.com]

Hi everyone!

I've just installed Qubes4 on a new X1 Carbon; BIOS is updated to 1.31.

I'm using 'sensors' to monitor temperature and fan speed.

I run like 5-7 qubes, no significant CPU usage shown in xentop.

Then the temperature is around 50-55 C and the fan is off.

When I now suspend by closing the lid and then resume, the following issues can occur:

Rarely: System does not wake up at all, neither by opening the lid nor by pressing the power button.

Sometimes: Wifi is not working after resume. Looking at the settings of sys-net shows that the network cards are not assigned to the vm. Trying to assign them is not possible, libvirt complains they are already assigned to sys-net and cannot be detached. Restarting sys-net fixes this.

Often: All usb devices disappear. 'qvm-usb' shows nothing. Restarting sys-usb fixes this.

Always: After resume the temperature idles at 60-70 C with the fan running at 4500RPM. xentop still shows no significant CPU usage. So this is an average 10 degrees hotter _with_ the fans actually running than it was before sleep. The only fix I know is rebooting the system. Apparently some power management feature is broken after resume. Even when I shutdown all qubes and only run dom0, temperature stays that high.

I primarily care for that last issue with the temperature. For the wifi issue I found some suggestions for workarounds I have yet to try.

But I did not find anything regarding the temperature issue. And it's the only one that can't be fixed live.

Now, what kind of logs can I provide? Also there are a number of settings in the BIOS that I don't fully understand. (Sleep mode of course is set to Linux.)

Does anyone else use an X1 carbon and might have some ideas? Any help is appreciated!

Ole

[rangu...@gmail.com]

Turn on Thunderbolt BIOS assist mode.
This resolves many buggy sleep issues in Linux and *BSD.
Note that some people have recently reported changing Thunderbolt BIOS assist as bricking their laptop BIOS.
Not me, though, and I've changed it back and forth many times.

[birdynam]

Hi,
Deactivation of the TPM resolves "suspends" issues on my lenovo x260.

Birdy

[tfm...@googlemail.com]

On Friday, October 26, 2018 at 8:45:49 PM UTC+2, rangu...@gmail.com wrote:
> Turn on Thunderbolt BIOS assist mode.

Thanks! In fact, it was already on. The BIOS states quite clearly that you should enable it when running Linux. So I never went without it.

[tfm...@googlemail.com]

On Friday, October 26, 2018 at 9:11:23 PM UTC+2, birdynam wrote:
> Deactivation of the TPM resolves "suspends" issues on my lenovo x260.

Thanks! I'm trying this now. It doesn't solve the temperature issue, though. But maybe it helps stabilize the other issues. Time will tell.

[tfm...@googlemail.com]

On Friday, October 26, 2018 at 3:14:14 PM UTC+2, tfm...@googlemail.com wrote:
> Always: After resume the temperature idles at 60-70 C with the fan running at 4500RPM. xentop still shows no significant CPU usage. So this is an average 10 degrees hotter _with_ the fans actually running than it was before sleep. The only fix I know is rebooting the system. Apparently some power management feature is broken after resume. Even when I shutdown all qubes and only run dom0, temperature stays that high.

Ok, apparently there is a deeper issue here: I noticed that I was capped at 400% CPU usage although this is a 4 core CPU with hyperthreading. 'lscpu' revealed that there was just one thread per core. Since I was absolutely sure that I previously could utilize 800%, I reinstalled Qubes.

I should mention that my system was Qubes 4 with 'qubes-dom0-update' run yesterday.

Now, with a fresh installation and no update, I have 2 threads per core again, and the above mentioned issue with increased heat after suspend is gone.

After running 'qubes-dom0-update' again and restarting, the threads per core are down to 1 again and the heat issue is back.

I guess I should report a bug regarding this.

PS: Can I change the title of the thread somehow, to better reflect the issue?

[Ranguvar]

No, as it is an email chain.

Threads per core being 1 actually makes sense given the recent Intel bugs. Hyperthreading is not secure.

I also have an X1C6 though, with no crazy CPU usage as of today after updating, rebooting, and suspending/resuming.

[tfm...@googlemail.com]

On Saturday, October 27, 2018 at 12:09:07 AM UTC+2, Ranguvar wrote:
> Threads per core being 1 actually makes sense given the recent Intel bugs. Hyperthreading is not secure.

So hyperthreading is deliberately disabled by the latest Qubes updates? Is this documented somewhere?

> I also have an X1C6 though, with no crazy CPU usage as of today after updating, rebooting, and suspending/resuming.

I don't have CPU usage. CPUs usage idles at < 10%. After resume simply the CPU temperature goes up 10 degrees. It is clearly noticeable as the fans are running more often and faster. This is actually how I noticed the issue. This does not happen without the Qubes update.

Also when I do 'xenpm get-cpufreq-para' after update, I get the data for CPU0 and CPU2, but I get errors '[CPU1] failed to get cpufreq parameter', same for CPU3.

Before update it showed params for CPU0 to CPU7.

So apparently this is not just hyperthreading being disabled. There seem to be troubles with two of the physical cores too.

So, if you have an X1C6 with updated Qubes too, do all these things work for you? Which BIOS version are you on?

[Ranguvar]

My bad. Reading comprehension is apparently hard.

I can replicate your CPU temp bug. Mine goes from ~42C on all cores to ~53C. 0% CPU usage, no fans.

xenpm also errors on two CPUs (1 and 3 for me).

BIOS v1.31, Hyperthreading on in BIOS

Hyperthreading disable documented here: https://groups.google.com/forum/m/#!topic/qubes-announce/JEtRYPmG-pAl

Cannot replicate using Arch with sway or Xorg 1.20.3, 4.18.16-ck kernel.

[tfm...@googlemail.com]

On Saturday, October 27, 2018 at 1:00:23 AM UTC+2, Ranguvar wrote:
> I can replicate your CPU temp bug. Mine goes from ~42C on all cores to ~53C. 0% CPU usage, no fans.
>
>
> xenpm also errors on two CPUs (1 and 3 for me).
>
>
> BIOS v1.31, Hyperthreading on in BIOS
>
>
> Hyperthreading disable documented here: https://groups.google.com/forum/m/#!topic/qubes-announce/JEtRYPmG-pAl
>
>
> Cannot replicate using Arch with sway or Xorg 1.20.3, 4.18.16-ck kernel.

Thanks alot! This is very helpful.

I re-enabled hyperthreading using the boot option 'smt=on'. All the CPU related issues are gone then, including the increased heat.

I hope the option to do that will be retained in the future. Hardened security is a great thing, but actually my prime reason for using Qubes is its superior organisation and ease to use.

Although I expect the heat issue to be a bug of some kind.

[tfm...@googlemail.com]

On Saturday, October 27, 2018 at 9:01:23 AM UTC+2, tfm...@googlemail.com wrote:
> On Saturday, October 27, 2018 at 1:00:23 AM UTC+2, Ranguvar wrote:
> > Hyperthreading disable documented here: https://groups.google.com/forum/m/#!topic/qubes-announce/JEtRYPmG-pAl

> I re-enabled hyperthreading using the boot option 'smt=on'. All the CPU related issues are gone then, including the increased heat.

If I have 'smt=on' but hyperthreading disabled in BIOS, the issues do not appear. I don't know if this is security-wise equivalent to 'smt=off'.

I guess I would take the risk for the added performance anyways. Although I would have to do some benchmarks first: My old system (Qubes 3.2) has hyperthreading disabled too since whenever this update was. I have not noticed any performance drops. In fact, I lately was less annoyed by its performance.

[tfm...@googlemail.com]

To be more precise regarding the wifi and USB issues:

One following issues can occur after suspend:

- All USB devices disappear from sys-usb, wifi is not working anymore. This can be fixed by restarting both sys-usb and sys-net.

- Directly after suspending, the speaker plays a loud, short melody (sounds a bit like the game-over-sound of an 80s computer game ^^). I guess this is some kind of diagnostic code? The LED shows sleep but the fan might keep running when it was running while suspending. The system cannot be woken up by opening the lid or pressing the power button.

The interesting part is: Either of these issues only seems to appear when a win7 HVM is running.

With only linux appvms running, I can close/open the lid 10 times (not that big of a sample, but still) without issues.

With a win7 HVM running, one of the issues happens everytime.

I don't have any other HVMs. But I guess this isn't predicated on win7 specifically.

[fiasc...@gmail.com]

Did you try the suspend/resume fix that DJB documented here: https://groups.google.com/d/msg/qubes-users/TmGDlkluJgM/1BFsQZWNDAAJ

It requires recompiling the kernel, but took me about 2 hours in total from reading, to understanding, to doing this in total on the same machine. It has worked for several of us (don't know anyone that it didn't work for). I have read that BIOS 1.31 should fix suspend/resume issues, but haven't tried it yet.

I haven't run a win7hvm or tried thermal issue fixes either.

[Ranguvar]

That patch forces S3 support on.

I've been using patches like that and patches to the DSDT so the laptop signals S3 for months.

It's not needed with the new BIOS that allows controlling that.

Should not play any role in this whatsoever.

I have not used HVMs.

[tfm...@googlemail.com]

On Saturday, October 27, 2018 at 8:57:43 AM UTC-4, tfm...@googlemail.com wrote:
> To be more precise regarding the wifi and USB issues:
>
> One following issues can occur after suspend:
>
> - All USB devices disappear from sys-usb, wifi is not working anymore. This can be fixed by restarting both sys-usb and sys-net.
>
> - Directly after suspending, the speaker plays a loud, short melody (sounds a bit like the game-over-sound of an 80s computer game ^^). I guess this is some kind of diagnostic code? The LED shows sleep but the fan might keep running when it was running while suspending. The system cannot be woken up by opening the lid or pressing the power button.
>
> The interesting part is: Either of these issues only seems to appear when a win7 HVM is running.

More precisely: If I just start an empty HVM (it will run a couple of seconds before it shuts down again) and suspend while it is running, the issue appears too.

On the other hand, after I installed qubes-windows-tools in my win7hvm, I can run that HVM and suspend without issues.

A bit of pseudo-science: On the surface, the difference would be that an HVM is shown as 'transient' all the time whereas with QWT the HVM will show proper states.

Anyways, it fixes the issue for me. So I guess I'm happy now. :-)

[ron]

O
> >
> > BIOS v1.31, Hyperthreading on in BIOS
> >
> >
> > Hyperthreading disable documented here: https://groups.google.com/forum/m/#!topic/qubes-announce/JEtRYPmG-pAl
> >
> >
> > Cannot replicate using Arch with sway or Xorg 1.20.3, 4.18.16-ck kernel.
>
> Thanks alot! This is very helpful.
>
> I re-enabled hyperthreading using the boot option 'smt=on'. All the CPU related issues are gone then, including the increased heat.
>
> I hope the option to do that will be retained in the future. Hardened security is a great thing, but actually my prime reason for using Qubes is its superior organisation and ease to use.

where exactly did you put this smt=on option in?
thanks
ron

[tfm...@googlemail.com]

In `/boot/efi/EFI/qubes/xen.cfg`. Look here https://www.qubes-os.org/news/2018/09/02/qsb-43/ for instructions and caveats.