Nvidia dGPU (MX150) Passthrough Issues
dylang...@gmail.com
I'm wanting to pass my dGPU down to an HVM appVM, so I can enjoy dGPU performance sometimes (Test Firefox's GPU Rendering Engine)
However I can't seem to start the appVM with the PCI Device Attached, I've set rd.qubes.hide_pci=01:00.0
Here's the Console Log:
https://pastebin.com/aKYNFSK8
And the Xen Console Log (Stubdom):
https://ufile.io/1ldov
(I had to upload it because it's 500KB worth of logs)
I have a feeling it has something to do with the stubdomain, but I'm not sure.
Any help would be appreciated, Cheers!
dylang...@gmail.com
```
[user@fedora-standalone ~]$ lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
00:03.0 VGA compatible controller: Device 1234:1111 (rev 02)
00:04.0 USB controller: Intel Corporation 82801DB/DBM (ICH4/ICH4-M) USB2 EHCI Controller (rev 10)
00:07.0 3D controller: NVIDIA Corporation GP108M [GeForce MX150] (rev a1)
```
dylang...@gmail.com
```
00:07.0 3D controller: NVIDIA Corporation GP108M [GeForce MX150] (rev a1)
Physical Slot: 7
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 44
Region 0: Memory at f4000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at e0000000 (64-bit, prefetchable) [size=256M]
Region 3: Memory at f0000000 (64-bit, prefetchable) [size=32M]
Region 5: I/O ports at c300 [size=128]
Capabilities: [60] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [78] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR+, OBFF Via message
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
AtomicOpsCtl: ReqEn-
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia_drm, nvidia
```
dmesg
```
[ 42.716142] nvidia: module license 'NVIDIA' taints kernel.
[ 42.760366] nvidia-nvlink: Nvlink Core is being initialized, major device number 239
[ 42.833842] nvidia-uvm: Loaded the UVM driver in 8 mode, major device number 237
```
I've installed the nvidia driver and disabled/blacklisted nouveau, however I can't use bumblebee due it complaining there's no Intel GPU (Using `00:03.0 VGA compatible controller: Device 1234:1111 (rev 02)`)
So it's there, I just can't tell applications to render on the dGPU, so close...
Has anyone been able to get this going?
Eric Duncan
> I've installed the nvidia driver and disabled/blacklisted nouveau, however I can't use bumblebee due it complaining there's no Intel GPU (Using `00:03.0 VGA compatible controller: Device 1234:1111 (rev 02)`)
>
My research led me to believe that you won't be able to use bumblebee with Qubes, as bumblebee expects both the Intel iGPU and Nvidia dGPU drivers in the same domain and Hybrid Graphics enabled in the BIOS. Since Qubes separates this by having the iGPU in dom0 (soon to be a new GUI domU?), and you are using pci-passthru dGPU to another domU, it won't have both.
I plan on experimenting with this as well; however, I already know that the dGPU will always have to be active in this setup - killing any battery savings.
What I don't know is if xen will put the dGPU will go into a power-savings mode if the domU it is attached to is shutdown. Or, better yet, if bumblebee can be installed in dom0 even with dGPU set as pci-passthru. I have my doubts; but, that is my plan to experiment with. My fallback is to disable the dGPU in the BIOS when on the road and switch back to Hybrid when only docked and only when I want to loadup AutoCAD or Blender (very rarely). However, I already know the X1 Exteme, and I'm guessing the P1, won't allow the Nvidia GPU to be disabled in the bios like earlier generation Thinkpads. Bleh, lots of workarounds.
> So it's there, I just can't tell applications to render on the dGPU, so close...
>
How can you tell "it's there"? Maybe look for it in the log such as:
# grep "X Driver" /var/log/Xorg.0.log
If you see something listed as Nvidia, then it is indeed loaded and could be used to offload hardware acceleration (such as Blender or perhaps AutoCAD shading).
What I don't know is if rendering 3D to the screen would work within Qubes.
OT: On my new Qubes/gaming desktop I am upgrading, I'll have all 4 displays connected to a single low-end GPU via HDMI inputs (monitors are currently all 1080p@120Hz). However, the 2x Vega 64s will connect via DVI (DP-to-DVI) to my primary 3 monitors for EyeFinity.
A trick I saw someone do with Xen on youtube was to "disconnect" the monitors from HDMI of the first GPU - this frees up their signal. Then, within another domU, enable the DP output and BAM, the monitors come back - dedicated to the gaming domU.
That should work with Qubes I believe with an HVM. However, I don't think that works with laptops though - unless you want to try dedicating the output of a specific external port to an external display. Maybe that might work? - keeping it disconnected from dom0, and enabling the output via xrandr within your nvidia dom0?
dylang...@gmail.com
> My fallback is to disable the dGPU in the BIOS when on the road
>
Yeah my Matebook X Pro has literally no settings relating to the dGPU, I can't turn it off from there.
It's my understanding, NVIDIA or any GPU really, has vBIOS, this is firmware that exists within the GPU and is a basic VGA "Adapter", then when you load the nvidia driver, that actually loads the 'real' nvidia firmware into the card, it then runs that firmware. I'm not too sure, but I think this is how it usually works, could be different on Mobile Chips (MX150), so _maybe_ when nothing is loaded, it's already in a 'sort of' low power state?
Apparently it's also possible to 'Power Off' the dGPU via ACPI calls (https://www.reddit.com/r/VFIO/comments/7d27sz/you_can_now_passthrough_your_dgpu_as_you_wish/dpvwka6/)
Bumblebee is super old, there should be another way to disable it, surely.
> How can you tell "it's there"?
>
Doing lspci -vvv, I can see the MX150 using the NVIDIA Driver, I tried catting that log, however it doesen't exist.
> What I don't know is if rendering 3D to the screen would work within Qubes.
>
I'm also skeptical, it's sort of passing through 2 iGPUs, you've got the Intel iGPU in dom0, then the virtual `VGA compatible controller: Device 1234:1111 (rev 02)` card within domU.
I can't see why it wouldn't work, you're just telling the kernel to offload 3D Rendering to the dGPU.
> OT: On my new Qubes/gaming desktop I am upgrading
>
I'm also doing this, but I'm thinking the Fedora/KVM route, I have a feeling Xen might be harder to setup.