AMDGPU/Nouveau on P550

[Matthew Croughan]

Just reaching out to see if anyone knows of a required patchset or extra Kconfig options to get amdgpu/nouveau working on https://github.com/sifive/riscv-linux/tree/dev/kernel/hifive-premier-p550 where I am using commit 18d8c8336d2946a7cb317ad2a4bc0a9eb0abb4f3

I hear Carl Perry might know, if they're on this mailing list

When an AMD GPU is plugged in (6700XT) and amdgpu is modprobe'd:

```

[root@p550:~]# lspci

00:00.0 PCI bridge: Beijing Eswin Computing Technology Co., Ltd. Device 2030 (rev 01)

01:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c5)

02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch

03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M/6850M XT] (rev c5)

03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller

[root@p550:~]# modprobe amdgpu

[   58.112626] [drm:amdgpu_device_init [amdgpu]] *ERROR* early_init of IP block <psp> failed -19

[   58.127782] [drm:amdgpu_device_init [amdgpu]] *ERROR* early_init of IP block <smu> failed -19

[   58.141479] [drm:dm_early_init [amdgpu]] *ERROR* DMUB firmware loading failed: -19

[   58.153111] [drm:amdgpu_device_init [amdgpu]] *ERROR* early_init of IP block <dm> failed -19

[   58.172005] [drm:amdgpu_device_init [amdgpu]] *ERROR* early_init of IP block <gfx_v10_0> failed -19

[   58.188811] [drm:amdgpu_device_init [amdgpu]] *ERROR* early_init of IP block <vcn_v3_0> failed -19

[   58.205506] amdgpu 0000:03:00.0: amdgpu: Fatal error during GPU init

```

When an Nvidia/Nouveau (RTX 3060) is plugged in and modprobe'd

```

[root@p550:~]# lspci
00:00.0 PCI bridge: Device 1fe1:2030 (rev 01)
01:00.0 VGA compatible controller: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GA106 High Definition Audio Controller (rev a1)

[root@p550:~]# rmmod nouveau

[root@p550:~]# modprobe nouveau
[  266.049389] genirq: Flags mismatch irq 12. 00000080 (nvkm) vs. 00000000 (ccache_ecc)
[Fri Jul  5 18:07:20 2024] genirq: Flags mismatch irq 12. 00000080 (nvkm) vs. 00000000 (ccache_ecc)
[Fri Jul  5 18:07:20 2024] nouveau: probe of 0000:01:00.0 failed with error -16

And on other occasions:

[    1.382830] nouveau 0000:01:00.0: Adding to iommu group 5
[    1.382892] nouveau 0000:01:00.0: enabling device (0000 -> 0003)
[    1.391069] nouveau: probe of 0000:01:00.0 failed with error -16

```

Cheers,

Matt

[Carl Perry]

I've been struggling to solve this problem. I made some headway last night, bit still have one critical kernel oops (down from 8) that's preventing newer GPUs from working. AMD RX400 and RX500 series cards work (GCN architecture based) work without issue, I've tested with an HD6450, RX240, RX550 and RX590. The newer cards (RX5000 and later, aka the rDNA architecture) are still giving me an error after fixing the ELF relocation calculations and enabling FPU support in the kernel. This may be a limitation or configuration error of the PCIe root port, or because of the memory model changes between RISC-V and other architectures. I'm still digging.

The nouveau situation is similar, but I'm having problems getting information out of the driver beyond negative error numbers like you have posted.

Let me gather logs, configs, and patches and I'll post a follow-up here later today. 

  - Carl

To unsubscribe from this group and stop receiving emails from it, send an email to devboard-commun...@riscv.org.

[Akira Tsukamoto gmail]

I had pretty much the same experience on my Radeon RX 5500 when I
tried to run it for this post.

https://x.com/AkiraTsukamoto/status/1855612233793097994?mx=2

I remember at the RISC-V Summit NA, sifive booth was running the p550
with RX560 or similar.

Akira

[Carl Perry]

Seems like you are a not on the correct branch? You probably want to be on https://github.com/sifive/riscv-linux/commits/rel/kernel/hifive-premier-p550/ and the latest commit is 022315aa62e5160face9de04d895090794d088b2 which is what I am using. You also want to make sure you have a recent release of linux-firmware installed and the correct decompressors in your kernel (I'll attach my kernel config as well).

That branch has the all import ELF Relocation Optimization patch, that will make many errors go away and improve system performance overall. I'm attaching the path I use for that which enables FPU support in the kernel for the AMDGPU driver on rDNA based cards. With those patches, I'm down to the following errors with an RX5500:

[   13.871647] amdgpu 0000:03:00.0: amdgpu: Fetched VBIOS from ROM BAR
[   13.879161] amdgpu: ATOM BIOS: xxx-xxx-xxx
[   13.943858] [drm] VCN decode is enabled in VM mode
[   13.949789] [drm] VCN encode is enabled in VM mode
[   13.972931] [drm] JPEG decode is enabled in VM mode
[   13.979258] amdgpu 0000:03:00.0: vgaarb: deactivate vga console
[   13.986542] amdgpu 0000:03:00.0: amdgpu: Trusted Memory Zone (TMZ) feature disabled as experimental (default)
[   13.997949] amdgpu 0000:03:00.0: amdgpu: PCIE atomic ops is not supported
[   14.006183] [drm] GPU posting now...
[   14.011294] [drm] vm size is 262144 GB, 4 levels, block size is 9-bit, fragment size is 9-bit
[   14.021236] amdgpu 0000:03:00.0: BAR 2: releasing [mem 0x8010000000-0x80101fffff 64bit pref]
[   14.031093] amdgpu 0000:03:00.0: BAR 0: releasing [mem 0x8000000000-0x800fffffff 64bit pref]
[   14.041059] pcieport 0000:02:00.0: BAR 15: releasing [mem 0x8000000000-0x8017ffffff 64bit pref]
[   14.051173] pcieport 0000:01:00.0: BAR 15: releasing [mem 0x8000000000-0x8017ffffff 64bit pref]
[   14.061348] pcieport 0000:00:00.0: BAR 15: releasing [mem 0x8000000000-0x8017ffffff 64bit pref]
[   14.071535] pcieport 0000:00:00.0: BAR 15: no space for [mem size 0x300000000 64bit pref]
[   14.081167] pcieport 0000:00:00.0: BAR 15: failed to assign [mem size 0x300000000 64bit pref]
[   14.091170] pcieport 0000:01:00.0: BAR 15: no space for [mem size 0x300000000 64bit pref]
[   14.100819] pcieport 0000:01:00.0: BAR 15: failed to assign [mem size 0x300000000 64bit pref]
[   14.110814] pcieport 0000:02:00.0: BAR 15: no space for [mem size 0x300000000 64bit pref]
[   14.120456] pcieport 0000:02:00.0: BAR 15: failed to assign [mem size 0x300000000 64bit pref]
[   14.130457] amdgpu 0000:03:00.0: BAR 0: no space for [mem size 0x200000000 64bit pref]
[   14.139839] amdgpu 0000:03:00.0: BAR 0: failed to assign [mem size 0x200000000 64bit pref]
[   14.149584] amdgpu 0000:03:00.0: BAR 2: no space for [mem size 0x00200000 64bit pref]
[   14.158872] amdgpu 0000:03:00.0: BAR 2: failed to assign [mem size 0x00200000 64bit pref]
[   14.168519] pcieport 0000:00:00.0: PCI bridge to [bus 01-03]
[   14.175625] pcieport 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[   14.183610] pcieport 0000:00:00.0:   bridge window [mem 0x41100000-0x412fffff]
[   14.192296] pcieport 0000:00:00.0: PCI bridge to [bus 01-03]
[   14.199399] pcieport 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[   14.207384] pcieport 0000:00:00.0:   bridge window [mem 0x41100000-0x412fffff]
[   14.216068] pcieport 0000:00:00.0:   bridge window [mem 0x8000000000-0x8017ffffff 64bit pref]
[   14.226065] pcieport 0000:01:00.0: PCI bridge to [bus 02-03]
[   14.233173] pcieport 0000:01:00.0:   bridge window [io  0x1000-0x1fff]
[   14.241162] pcieport 0000:01:00.0:   bridge window [mem 0x41100000-0x411fffff]
[   14.249841] pcieport 0000:01:00.0:   bridge window [mem 0x8000000000-0x8017ffffff 64bit pref]
[   14.259845] pcieport 0000:02:00.0: PCI bridge to [bus 03]
[   14.266681] pcieport 0000:02:00.0:   bridge window [io  0x1000-0x1fff]
[   14.274674] pcieport 0000:02:00.0:   bridge window [mem 0x41100000-0x411fffff]
[   14.283351] pcieport 0000:02:00.0:   bridge window [mem 0x8000000000-0x8017ffffff 64bit pref]
[   14.293376] [drm] Not enough PCI address space for a large BAR.
[   14.293385] amdgpu 0000:03:00.0: BAR 0: assigned [mem 0x8000000000-0x800fffffff 64bit pref]
[   14.310555] amdgpu 0000:03:00.0: BAR 2: assigned [mem 0x8010000000-0x80101fffff 64bit pref]
[   14.320385] amdgpu 0000:03:00.0: amdgpu: VRAM: 8176M 0x0000008000000000 - 0x00000081FEFFFFFF (8176M used)
[   14.331408] amdgpu 0000:03:00.0: amdgpu: GART: 512M 0x0000000000000000 - 0x000000001FFFFFFF
[   14.341242] amdgpu 0000:03:00.0: amdgpu: AGP: 267894784M 0x0000008400000000 - 0x0000FFFFFFFFFFFF
[   14.351498] [drm] Detected VRAM RAM=8176M, BAR=256M
[   14.357811] [drm] RAM width 128bits GDDR6
[   14.363576] [drm] amdgpu: 8176M of VRAM memory ready
[   14.370018] [drm] amdgpu: 8019M of GTT memory ready.
[   14.376209] [drm] GART: num cpu pages 131072, num gpu pages 131072
[   14.383864] [drm] PCIE GART of 512M enabled (table at 0x0000008000000000).
[   14.403007] [drm] Found VCN firmware Version ENC: 1.23 DEC: 8 VEP: 0 Revision: 1
[   14.410741] amdgpu 0000:03:00.0: amdgpu: Will use PSP to load VCN firmware
[   14.595967] [drm] reserve 0x900000 from 0x81fd000000 for PSP TMR
[   14.649179] amdgpu 0000:03:00.0: amdgpu: RAS: optional ras ta ucode is not available
[   14.666194] amdgpu 0000:03:00.0: amdgpu: RAP: optional rap ta ucode is not available
[   14.674777] amdgpu 0000:03:00.0: amdgpu: SECUREDISPLAY: securedisplay ta ucode is not available
[   14.685005] amdgpu 0000:03:00.0: amdgpu: use vbios provided pptable
[   14.692673] amdgpu 0000:03:00.0: amdgpu: smc_dpm_info table revision(format.content): 4.5
[   14.735721] amdgpu 0000:03:00.0: amdgpu: SMU is initialized successfully!
[   14.742873] [drm] Display Core failed to initialize with v3.2.247!
[   14.750441] Unable to handle kernel access to user memory without uaccess routines at virtual address 0000000000000008
[   14.762614] Oops [#1]
[   14.762618] Modules linked in: amdgpu(+) binfmt_misc snd_hda_codec_hdmi amdxcp drm_exec mfd_core snd_hda_intel snd_intel_dspcfg snd_soc_audio_graph_card gpu_sched snd_hda_codec drm_buddy drm_suballoc_helper snd_hwdep drm_ttm_helper ttm snd_hda_core backlight nls_iso8859_1 snd_soc_simple_card_utils pvrsrvkm at24 pac193x usbmouse ina2xx esw_es8328_codec gpio_dwapb snd_soc_i2s gpio_generic snd_soc_hdmi_codec dwmac_eic7700 regmap_i2c stmmac_platform snd_soc_core fusb303b stmmac snd_pcm_dmaengine typec snd_pcm pcs_xpcs dw_wdt phylink snd_timer input_leds dw_axi_dmac_platform virt_dma snd soundcore evbug joydev usbkbd uio_pdrv_genirq uio sch_fq_codel crypto_user nfnetlink ip_tables x_tables xfs(E) libcrc32c(E) hid_generic(E) usbhid(E) hid(E) spi_eswin_bootspi(E)
[   14.835247] CPU: 3 PID: 339 Comm: (udev-worker) Tainted: G            E      6.6.21-edolnx-6.6.21-1 #1
[   14.835254] Hardware name: SiFive HiFive Premier P550 (DT)
[   14.835258] epc : drm_atomic_private_obj_fini+0x16/0x60
[   14.858231]  ra : amdgpu_dm_fini+0x3e/0x1d6 [amdgpu]
[   14.868845] epc : ffffffff807119be ra : ffffffff02b33710 sp : ffff8f8000a934f0
[   14.868851]  gp : ffffffff81b8fd58 tp : ffffaf808661d580 t0 : 0000000000000000
[   14.868856]  t1 : 0000000000000000 t2 : 736944205d6d7264 s0 : ffff8f8000a93510
[   14.868860]  s1 : 0000000000040000 a0 : ffffaf80920bf638 a1 : 0000000000000000
[   14.900772]  a2 : 0000000000000100 a3 : 0000000000000122 a4 : 0000000000000000
[   14.900777]  a5 : 0000000000000000 a6 : 0000000000000000 a7 : 0000000000000000
[   14.900780]  s2 : ffffaf8092080010 s3 : ffffaf8092080000 s4 : ffffaf80920bf000
[   14.900784]  s5 : 0000000000000001 s6 : ffff8f8000a935a8 s7 : 0000000000000004
[   14.900788]  s8 : 0000000000000002 s9 : 0000000000000001 s10: 0000000000000005
[   14.900792]  s11: ffffaf8092080000 t3 : 0000000000000000 t4 : 0000000000000000
[   14.948664]  t5 : 0000000000000000 t6 : ffff8f8000a93348
[   14.948667] status: 0000000200000120 badaddr: 0000000000000008 cause: 000000000000000f
[   14.948672] [<ffffffff807119be>] drm_atomic_private_obj_fini+0x16/0x60
[   14.948691] [<ffffffff02b33710>] amdgpu_dm_fini+0x3e/0x1d6 [amdgpu]
[   14.980521] [<ffffffff02b3e770>] amdgpu_dm_init.isra.0+0x154/0x1a26 [amdgpu]
[   14.993222] [<ffffffff02b40056>] dm_hw_init+0x14/0x2a [amdgpu]
[   15.001903] [<ffffffff028fa04e>] amdgpu_device_init+0x140c/0x1e72 [amdgpu]
[   15.011638] [<ffffffff028fc2a2>] amdgpu_driver_load_kms+0x1e/0x154 [amdgpu]
[   15.021460] [<ffffffff028f38a0>] amdgpu_pci_probe+0xf6/0x32a [amdgpu]
[   15.030756] [<ffffffff80638862>] pci_device_probe+0x8c/0x158
[   15.037862] [<ffffffff8078a9bc>] really_probe+0x8c/0x33e
[   15.044617] [<ffffffff8078acd8>] __driver_probe_device+0x6a/0x124
[   15.052161] [<ffffffff8078adbe>] driver_probe_device+0x2c/0xbc
[   15.059443] [<ffffffff8078afd2>] __driver_attach+0xaa/0x1cc
[   15.066461] [<ffffffff807888b6>] bus_for_each_dev+0x62/0xba
[   15.073480] [<ffffffff8078a3ae>] driver_attach+0x1a/0x28
[   15.080236] [<ffffffff80789b3c>] bus_add_driver+0x138/0x22a
[   15.087255] [<ffffffff8078c04c>] driver_register+0x40/0xf2
[   15.094186] [<ffffffff806371f2>] __pci_register_driver+0x44/0x5a
[   15.101645] [<ffffffff02763070>] amdgpu_init+0x70/0x1000 [amdgpu]
[   15.110590] [<ffffffff8000293a>] do_one_initcall+0x56/0x246
[   15.117607] [<ffffffff800cc846>] do_init_module+0x88/0x1fa
[   15.124539] [<ffffffff800ce382>] load_module+0x16f6/0x1b6a
[   15.131470] [<ffffffff800cea0a>] init_module_from_file+0x78/0xc6
[   15.138926] [<ffffffff800cec20>] __riscv_sys_finit_module+0x1c8/0x2e6
[   15.146821] [<ffffffff80bf280e>] do_trap_ecall_u+0x4a/0x126
[   15.153840] [<ffffffff80bfe2bc>] ret_from_exception+0x0/0x64
[   15.160954] Code: e822 e426 ec06 1000 651c 6118 0613 1000 0693 1220 (e71c) e398
[   15.169859] ---[ end trace 0000000000000000 ]---

I'll try the nouveau cards again by EOW.

  -Carl

[Akira Tsukamoto gmail]

Hi Carl,

Do you happen to know the Ubuntu image comes with the latest p550
would have the latest commit of
022315aa62e5160face9de04d895090794d088b2 with a recent release of
linux-firmware,
which announced by the email "The HiFive Premier P550 is Back- Now
with Ubuntu and at a Lower Price"?

Or where could get the Ubuntu image of it?

Cheers,

Akira

[Carl Perry]

The kernel commit is tagged as the Ubuntu release. I don't know if the image is available yet, but it's pretty easy to make one. I'm currently running Arch off a SATA SSD, but using debootstrap to get Ubuntu installed is pretty simple as well.

  -Carl

[Akira Tsukamoto gmail]

Hi Carl,

This is a different subject than the kernel but, does the p550 have a
way to boot from a fast storage such as M2.SSD or so?

Akira

[Samuel Holland]

Hi Carl,

On 12/11/24 16:17, 'Carl Perry' via RISC-V Developer Board Community wrote:
> Seems like you are a not on the correct branch? You probably want to be
> on https://github.com/sifive/riscv-linux/commits/rel/kernel/hifive-
> premier-p550/ and the latest commit is
> 022315aa62e5160face9de04d895090794d088b2 which is what I am using. You
> also want to make sure you have a recent release of linux-firmware
> installed and the correct decompressors in your kernel (I'll attach my
> kernel config as well).
>
> That branch has the all import ELF Relocation Optimization patch, that
> will make many errors go away and improve system performance overall.
> I'm attaching the path I use for that which enables FPU support in the
> kernel for the AMDGPU driver on rDNA based cards. With those patches,
> I'm down to the following errors with an RX5500:
>

> ...

> [   14.735721] amdgpu 0000:03:00.0: amdgpu: SMU is initialized successfully!
> [   14.742873] [drm] Display Core failed to initialize with v3.2.247!

You don't have kernel-mode FPU support enabled properly. That one commit
isn't enough. You need at least patches 1, 11, 12, and 13 from that
series[1]. If you get it right, your .config will have this line in it:

CONFIG_DRM_AMD_DC_FP=y

Here's a branch where I backported these patches to the HiFive Premier
P550 kernel[2]. These changes should eventually be included in the
board's official kernel builds.

Regards,
Samuel

[1]:
https://lore.kernel.org/linux-riscv/20240329072441.5914...@sifive.com/
[2]:
https://github.com/sifive/riscv-linux/commits/dev/sholland/hifive-premier-p550-amdgpu

[Carl Perry]

Yes, with some major caveats: like most boards that have come out recently all the EIC7700 based boards have eMMC, and that is where the SPL, OpenSBI, and U-Boot live. You can also boot of the SD card slot. You can't boot off SATA or NVMe directly. Unlike many of the recent boards, the U-Boot is extremely modern and has most of it's options turned on (including extlinux support, SATA support, and NVMe support). So my system has a small ext2 boot partition on the eMMC which then can pull kernels and initrds from the SATA SSD. The other major problem is the PCIe interface: the EIC7700 only has a single PCIe root port with 4 lanes. On the P550 it's wired to the x16 mechanical slot (it's still only x4 electrical). You can put an M.2 adapter in there to use an NVMe drive, but then you limited to a single PCIe device so that precludes having a GPU as well. The Megrez board seems to have the single port wired to both an M.2 slot and a x4 Mechanical and Electrical slot so you can choose to use either (but not both). So if fast storage is your desire, it's possible.

  -Carl

[Carl Perry]

Thank you for that update Samuel! I will point my packages at your kernel tomorrow and try again. Looking forward to sharing the results.

  -Carl

[Matthew Croughan]

Yes, the reason I am using the dev branch instead of the rel branch is because of a compiler error relating to the IMG_VOLCANIC stuff that's in the p550 defconfig, this error does not exist on the dev branch. But I can just disable the volcanic GPU for now in order to try your suggestions.

In file included from ../include/linux/string.h:294,
                 from ../include/linux/bitmap.h:11,
                 from ../include/linux/cpumask.h:12,
                 from ../include/linux/smp.h:13,
                 from ../include/linux/lockdep.h:14,
                 from ../include/linux/mutex.h:17,
                 from /build/source/drivers/gpu/drm/img/img-volcanic/include/lock_types.h:53,
                 from /build/source/drivers/gpu/drm/img/img-volcanic/services/shared/include/lock.h:50,
                 from /build/source/drivers/gpu/drm/img/img-volcanic/services/shared/include/sync_internal.h:52,
                 from /build/source/drivers/gpu/drm/img/img-volcanic/services/server/include/device.h:54,
                 from ../drivers/gpu/drm/img/img-volcanic/services/server/devices/volcanic/rgxdevice.h:53,
                 from ../drivers/gpu/drm/img/img-volcanic/services/server/devices/volcanic/rgxdebug.c:48:
In function 'strlcpy',
    inlined from '_FillAppForFWFaults' at ../drivers/gpu/drm/img/img-volcanic/services/server/devices/volcanic/rgxdebug.c:1126:5,
    inlined from '_RecordFaultInfo' at ../drivers/gpu/drm/img/img-volcanic/services/server/devices/volcanic/rgxdebug.c:1281:7,
    inlined from '_RGXDumpFWHWRInfo' at ../drivers/gpu/drm/img/img-volcanic/services/server/devices/volcanic/rgxdebug.c:2154:7:
../include/linux/fortify-string.h:255:25: error: call to '__write_overflow' declared with attribute error: detected write beyond size of object (1st parameter)
  255 |                         __write_overflow();
      |                         ^~~~~~~~~~~~~~~~~~

[Carl Perry]

Using Samuel's branch, I can confirm that every Radeon card I have is now working:

radeon driver tested with HD6450

amdgpu driver tested with RX240 RX550 RX590 RX5500 RX6600

I've decided to make two variants of the kernel, one with the iGPU support enabled and one without. The DGPUs work with the iGPU enabled but there are weird quirks: Wayland is very unstable, there are phantom monitors, you never get a text console on the DGPU, etc. Sadly the drivers for the ESWIN DRM driver and the IMG GPU cannot be built as modules, hence why I've decided to split into two drivers. I'll be posting both to GitHub later today and I'll reply here with links once I do. I just need the DGPU only kernel to compile and then run some quick tests.

The other thing I am going to try and debug over the weekend is nouveau driver support. I've bumped the logging levels so hopefully I can get more insight other than "Error -19". I'll keep everyone posted.

  -Carl

[Carl Perry]

Here are my repos with configs and pointers to the source trees. This first one enables various dGPUs and the  iGPU:

https://github.com/edolnx/linux-edolnx-sifive-p550

And this one does not enable the iGPU, and only supports dGPUs:

https://github.com/edolnx/linux-edolnx-sifive-p550-dgpu

I've built the configs so that they should be friendly to most distros. I'm building on Arch, but it has a lot of the necessary options enabled to be used on Fedora as well (mostly xz initramfs and firmware support).

Feedback welcome, still working on nouveau/nvidia compatibility and getting the text console working (which is being very stubborn for some reason).

  -Carl

[Han Gao/Revy]

CONFIG_ESWIN_VIRTUAL_DISPLAY=y

Manually disabling this option can delete non-existent virtual display devices.

Sync the following changes so that you don't have to disable this option manually through debugfs control

https://github.com/eswincomputing/linux-stable/commit/7d4b017ade19a9abbc191807fb3696c06b6d733b