OpenCL with Radeon GPU

[Arno Lehmann]

Hi all,

I'm currently trying to set up a new workstation and wanted to enable
OpenCL for future use.

GPU hardware is an AMD Radeon RX 6750 XT.

I have a base installation of Debian 12 up and running, kernel 6.1.0.7
including amdgpu and Mesa 22.3.

Any attempt to actually run any OpenCL program results in an error:

fatal error: cannot open file
'/usr/lib/clc/gfx1030-amdgcn-mesa-mesa3d.bc': No such file or directory

which is depending on the using program reported more or less easily
visible. clinfo is sufficient to show this.

Trying to diagnose this with available documentation I learned that
- I am not even able to determine if OpenCL itself should work,
- if it should work with the hardware I have here,
- which versions of amdgpu, xserver, Mesa, llvm or clang I need or which
particular drivers / packages to install

I have tried several versions of the possible packages, but lacking
documentation that makes sense to me that was more like blindly poking
in the dark.

I even tried following some advice to use AMD's out-of-kernel driver
pack, but the installation script stumbles over dependencies (which may
or may not be due to the not-yet-stable base distro).

Anyway, the current state is less than satisfying, because I don't even
understant if OpenCL with Radeon RX 6xxx GPU can even work on Linux.

I would give a different distro a try, but as a mostly-Debian-person I'm
not really inclined to start learning to maintain more distros. Just
using Windows would be a solution I would not appreciate at all...

So, my question here: Can anybody confirm this can work in general, or
even point me to a place for current, correct documentation?

Alternatively, a pointer to where I short report those problems --
Debian bug (which package?), freedesktop.org's mesa, Linux kernel
developers would be appreciated.


Thanks,

Arno


--
Arno Lehmann

IT-Service Lehmann
Sandstr. 6, 49080 Osnabrück

[Sarunas Burdulis]

Yes, in general OpenCL for current AMD GPUs works with the open source
Linux kernel's amdgpu.ko module. On Ubuntu one can install ROCm from
AMD's repositories

`apt install rocm-opencl-runtime`

which is open source well, but not in standard repositories. Not sure if
that extends to Debian.

I don't know if this can be substituted in any degree by Mesa.

--
Sarunas Burdulis
Dartmouth Mathematics
math.dartmouth.edu/~sarunas

· https://useplaintext.email ·

[Timothy M Butterworth]

Debian has: `sudo apt info rocm-device-libs` which provides openCL runtime.

 

I don't know if this can be substituted in any degree by Mesa.

--
Sarunas Burdulis
Dartmouth Mathematics
math.dartmouth.edu/~sarunas

· https://useplaintext.email ·

--

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/
⠈⠳⣄⠀⠀

[Arno Lehmann]

Hi Timothy,

Am 24.04.2023 um 23:38 schrieb Timothy M Butterworth:
>
>
> On Mon, Apr 24, 2023 at 4:00 PM Sarunas Burdulis
> <sar...@math.dartmouth.edu <mailto:sar...@math.dartmouth.edu>> wrote:
>
> On 4/24/23 14:02, Arno Lehmann wrote:

...

> Yes, in general OpenCL for current AMD GPUs works with the open source
> Linux kernel's amdgpu.ko module. On Ubuntu one can install ROCm from
> AMD's repositories
>
> `apt install rocm-opencl-runtime`
>
> which is open source well, but not in standard repositories. Not
> sure if
> that extends to Debian.
>
>
> Debian has: `sudo apt info rocm-device-libs` which provides openCL runtime.

That did not change things, clinfo still causes the
=== CL_PROGRAM_BUILD_LOG ===

fatal error: cannot open file
'/usr/lib/clc/gfx1030-amdgcn-mesa-mesa3d.bc': No such file or directory

message which seems to indicate that some devie specific stuff is missing.

I'll try to install the Ubunto packages later as this may require some
serious cleanup first (and sorry for replying to you directly earlier...
I guess after this days work, hitting the "Reply to List" button was a
bit beyond my capabilities).

Thanks for your suggestions!

Cheers,

Arno

> I don't know if this can be substituted in any degree by Mesa.
>
> --
> Sarunas Burdulis
> Dartmouth Mathematics

> math.dartmouth.edu/~sarunas <http://math.dartmouth.edu/~sarunas>
>
> · https://useplaintext.email <https://useplaintext.email> ·

>
>
>
> --
> ⢀⣴⠾⠻⢶⣦⠀
> ⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system

> ⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/ <https://www.debian.org/>
> ⠈⠳⣄⠀⠀

[Yvan Masson]

Le 24/04/2023 à 23:57, Arno Lehmann a écrit :
> Hi Timothy,
>
> Am 24.04.2023 um 23:38 schrieb Timothy M Butterworth:
>>
>>
>> On Mon, Apr 24, 2023 at 4:00 PM Sarunas Burdulis
>> <sar...@math.dartmouth.edu <mailto:sar...@math.dartmouth.edu>> wrote:
>>
>>     On 4/24/23 14:02, Arno Lehmann wrote:
> ...
>>     Yes, in general OpenCL for current AMD GPUs works with the open
>> source
>>     Linux kernel's amdgpu.ko module. On Ubuntu one can install ROCm from
>>     AMD's repositories
>>
>>     `apt install rocm-opencl-runtime`
>>
>>     which is open source well, but not in standard repositories. Not
>>     sure if
>>     that extends to Debian.
>>
>>
>> Debian has: `sudo apt info rocm-device-libs` which provides openCL
>> runtime.
>
> That did not change things, clinfo still causes the
> === CL_PROGRAM_BUILD_LOG ===
> fatal error: cannot open file
> '/usr/lib/clc/gfx1030-amdgcn-mesa-mesa3d.bc': No such file or directory
>
> message which seems to indicate that some devie specific stuff is missing.

I don't know anything about this topic, but for such a issue the first
thing I do is to search inside packages contents from this page:
https://www.debian.org/distrib/packages

In you case, the results might indicate that your GPU is not supported
by Debian ?
https://packages.debian.org/search?searchon=contents&keywords=mesa3d.bc&mode=path&suite=stable&arch=any

[Jörg-Volker Peetz]

Hi,

which mesa packages do you have installed?
In Debian testing I see the package `mesa-opencl-icd`.
I have no experience with opencl and AMD graphic cards but are very interested
how it works in Debian.

Regards,
Jörg.

[Arno Lehmann]

Hi Yvan,

Am 25.04.2023 um 11:06 schrieb Yvan Masson:
> Le 24/04/2023 à 23:57, Arno Lehmann a écrit :
>> Hi Timothy,
>>
>> Am 24.04.2023 um 23:38 schrieb Timothy M Butterworth:
>>>
>>>
>>> On Mon, Apr 24, 2023 at 4:00 PM Sarunas Burdulis
>>> <sar...@math.dartmouth.edu <mailto:sar...@math.dartmouth.edu>> wrote:
>>>
>>>     On 4/24/23 14:02, Arno Lehmann wrote:
>> ...
>>>     Yes, in general OpenCL for current AMD GPUs works with the open
>>> source
>>>     Linux kernel's amdgpu.ko module.

...

> I don't know anything about this topic, but for such a issue the first
> thing I do is to search inside packages contents from this page:
> https://www.debian.org/distrib/packages

I did that, but there are quite some assumptions implied by this
approach, mostly that a file of that particular name is *required* for
nay OpenCL implementation with this hardware -- and that is something I
hesitated to believe in :-)

> In you case, the results might indicate that your GPU is not supported
> by Debian ?
> https://packages.debian.org/search?searchon=contents&keywords=mesa3d.bc&mode=path&suite=stable&arch=any
>>
>> I'll try to install the Ubunto packages later as this may require some

Always this same typo of mine ^
This is either the reason for, or the result of, me not liking Ubuntu ;-)

>> serious cleanup first

Well, I did some of that cleanup (essentially it was removing the
relevant mesa packages, purging the configuration (removing the entries
in the /etc/OpenCL/vendors directory) and leaving the one from the AMD
packages behind.

The results were most exciting:
- clinfo had no error messages any more
- I could start OpenCL Primegrid tasks via boinc

Definitely a step forward.

Unfortunately there are more steps ahead, as any Primegrid task using
the GPU fails with an error:

opencl error: CL_OUT_OF_HOST_MEMORY

As I have no other real OpenCL workload I can try with any reliability,
and I also have lots of host memory available, I suspect this due to the
messy state of the software or its installation. And, as the
installation has definitely not been done in any documented manner, I
would not be surprised if that's the problem.

So, the challenge remains to find a way that reliably provides a working
system, but it seems as though Mesa is not going to be the way towards
this solution.

Thanks for all your input so far!

Arno

[Arno Lehmann]

Hi Jörg-Volker,

Am 25.04.2023 um 11:19 schrieb Jörg-Volker Peetz:
> Hi,
>
> which mesa packages do you have installed?

None providing OpenCL capabilities at that time, but if you followed
this thread, you are aware of this already:

# LANG=C dpkg -l '*mesa*' | cut -c 1-72
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Tri
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Archite
+++-===============================-============================-=======
ii libegl-mesa0:amd64 22.3.6-1+deb12u1 amd64
un libegl1-mesa-dev <none> <none>
un libgl1-mesa-dev <none> <none>
ii libgl1-mesa-dri:amd64 22.3.6-1+deb12u1 amd64
un libgl1-mesa-glx <none> <none>
ii libglapi-mesa:amd64 22.3.6-1+deb12u1 amd64
un libgles2-mesa-dev <none> <none>
ii libglu1-mesa:amd64 9.0.2-1.1 amd64
ii libglx-mesa0:amd64 22.3.6-1+deb12u1 amd64
un libwayland-egl1-mesa <none> <none>
ii mesa-amdgpu-common-dev 1:22.3.0.50403-1538762.20.04 amd64
ii mesa-amdgpu-omx-drivers:amd64 1:22.3.0.50403-1538762.20.04 amd64
ii mesa-amdgpu-va-drivers:amd64 1:22.3.0.50403-1538762.20.04 amd64
ii mesa-amdgpu-vdpau-drivers:amd64 1:22.3.0.50403-1538762.20.04 amd64
ii mesa-common-dev:amd64 22.3.6-1+deb12u1 amd64
un mesa-opencl-icd <none> <none>
ii mesa-utils 8.5.0-1 amd64
ii mesa-utils-bin:amd64 8.5.0-1 amd64
un mesa-utils-extra <none> <none>
ii mesa-va-drivers:amd64 22.3.6-1+deb12u1 amd64
ii mesa-vdpau-drivers:amd64 22.3.6-1+deb12u1 amd64
ii mesa-vulkan-drivers:amd64 22.3.6-1+deb12u1 amd64
un mesag3 <none> <none>
un xlibmesa3 <none> <none>

> In Debian testing I see the package `mesa-opencl-icd`.
> I have no experience with opencl and AMD graphic cards but are very
> interested how it works in Debian.

That's more or less where I am myself... seems a bit too complex for me,
at this time ;-)

As I trust the documentation in general, I am kind of confident that the
problems I see derive from my usage of a too recent GPU; the older
Radeon Architectures probably works out of the box.

The wiki page https://wiki.debian.org/AtiHowTo would deserve a bit of an
overhaul, I think, seeing that it does not mention OpenCL at all, for
example. If I just knew what to put there, I'd be happy to suggest
something...

Best results so far are with the AMD vendor drivers which are *not*
intended to be used on Debian:

# LANG=C dpkg -l '*rocm*' | cut -c 1-72
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Tri
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=====================-=====================-============-===========
un procmail <none> <none> (no descrip
ii rocm-core 5.4.3.50403-121~20.04 amd64 Radeon Open
ii rocm-device-libs 1.0.0.50403-121~20.04 amd64 Radeon Open
ii rocm-language-runtime 5.4.3.50403-121~20.04 amd64 Radeon Open
ii rocm-ocl-icd 2.0.0.50403-121~20.04 amd64 opencl buil
ii rocm-opencl 2.0.0.50403-121~20.04 amd64 opencl buil
ii rocm-opencl-runtime 5.4.3.50403-121~20.04 amd64 Radeon Open


Cheers,

Arno

> Regards,
> Jörg.

[Arno Lehmann]

Hi all,

a bit of research resulted in me finding this (old) page, which
indicated there were some environment variables I could use to control
memory allocation of AMD's OpenCL implementation:

https://community.amd.com/t5/drivers-software/solved-clinfo-reports-error-33-of-quot-global-free-memory-amd/td-p/172760

So I decided to give that a try.

First step: Ensure I can run an OpenCL program from the shell. I used
the Primegrid binary for that:

#
/var/lib/boinc-client/projects/www.primegrid.com/genefer22g_linux64_22.12.02
-h
geneferg version 22.12.2 (linux x64, gcc-7.5.0, boinc-7.20.2)
Copyright (c) 2022, Yves Gallot
genefer is free source code, under the MIT license.

Command line: '-h'

Running on device 'gfx1031', vendor 'Advanced Micro Devices, Inc.',
version 'OpenCL 2.0 ', driver '3513.0 (HSA1.1,LC)'.
330000000^{2^15} + 1: 00:00:35, 0.0386 ms/bit, data size: 1.12 MB.
200000000^{2^16} + 1: 00:01:28, 0.0489 ms/bit, data size: 2.25 MB.
120000000^{2^17} + 1: 00:04:35, 0.0784 ms/bit, data size: 4.5 MB.
18000000^{2^18} + 1: 00:14:01, 0.133 ms/bit, data size: 9 MB.
5500000^{2^19} + 1: 00:49:15, 0.252 ms/bit, data size: 18 MB.
2000000^{2^20} + 1: 01:58:46, 0.325 ms/bit, data size: 24 MB.
910000^{2^21} + 1: 07:04:18, 0.613 ms/bit, data size: 48 MB.
270000^{2^22} + 1: 25:34:55, 1.22 ms/bit, data size: 96 MB.
1000000^{2^22} + 1: 30:51:50, 1.33 ms/bit, data size: 96 MB.
500000^{2^23} + 1: 116:24:31, 2.64 ms/bit, data size: 192 MB.


So that worked.

It also did not cause an error, but of course, test data != real
workload, right?

Next step Ensure I can reproduce the error:

#
/var/lib/boinc-client/projects/www.primegrid.com/genefer22g_linux64_22.12.02
-p -n 22 -b 1053460 -f gproof
geneferg version 22.12.2 (linux x64, gcc-7.5.0, boinc-7.20.2)
Copyright (c) 2022, Yves Gallot
genefer is free source code, under the MIT license.

Command line: '-p -n 22 -b 1053460 -f gproof'

Running on device 'gfx1031', vendor 'Advanced Micro Devices, Inc.',
version 'OpenCL 2.0 ', driver '3513.0 (HSA1.1,LC)', data size: 96 MB.
0.0202% done, 28:15:37 remaining, 1.21 ms/bit.


Interesting, this seems to work without problem. Right now, I'm at

7.29% done, 26:21:57 remaining, 1.22 ms/bit.


which is much longer than what I've seen before.

My conclusion for now: The boinc service must have some limits set.

systemctl show gives me, among others:

LimitCPU=infinity
LimitCPUSoft=infinity
LimitFSIZE=infinity
LimitFSIZESoft=infinity
LimitDATA=infinity
LimitDATASoft=infinity
LimitSTACK=infinity
LimitSTACKSoft=8388608
LimitCORE=infinity
LimitCORESoft=0
LimitRSS=infinity
LimitRSSSoft=infinity
LimitNOFILE=524288
LimitNOFILESoft=1024
LimitAS=infinity
LimitASSoft=infinity
LimitNPROC=253399
LimitNPROCSoft=253399
LimitMEMLOCK=8388608
LimitMEMLOCKSoft=8388608
LimitLOCKS=infinity
LimitLOCKSSoft=infinity
LimitSIGPENDING=253399
LimitSIGPENDINGSoft=253399
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=infinity
LimitRTTIMESoft=infinity


My first candidate would be LimitMEMLOCK as I suspect that, for
interaction between GPU and CPU, shared and locked would be a likely
way. (You notice I know nearly nothing of OpenCL...)


I do know how 'systemctl edit' works, though, and set the limit to 1 GB:

LimitMEMLOCK=1073741824
LimitMEMLOCKSoft=1073741824

Which did not help, same error after similar time.

Still, seeing that I can run the binary in question from the shell, I'm
kind of confident this should be solvable via proper unit configuration.


Which leaves me with one question for this mail thread: Can anybody
recommend a test program for OpenCL functionality?


Thanks,

[Jörg-Volker Peetz]

Hi Arno,

I'm wondering if package mesa-opencl-icd would do the trick (version
22.3.6-1+deb12u1 like the other installed mesa packages). From the description
it seems to be the right package:

free implementation of the OpenCL API -- ICD runtime
This package contains the mesa implementation of the OpenCL (Open Compute
Language) library, which is intended for use with an ICD loader. OpenCL provides
a standardized interface for computational analysis on graphical processing units.

Regards,
Jörg.

<snip>

[Arno Lehmann]

Hi Jörg-Volker,

On 26.04.23 at 11:34, Jörg-Volker Peetz wrote:
> Hi Arno,
>
> Arno Lehmann wrote on 25/04/2023 12:54:
>> Hi Jörg-Volker,
>>
>> Am 25.04.2023 um 11:19 schrieb Jörg-Volker Peetz:

...

>>> In Debian testing I see the package `mesa-opencl-icd`.
>>> I have no experience with opencl and AMD graphic cards but are very
>>> interested how it works in Debian.
>>
>> That's more or less where I am myself... seems a bit too complex for
>> me, at this time ;-)
>
> I'm wondering if package mesa-opencl-icd would do the trick (version
> 22.3.6-1+deb12u1 like the other installed mesa packages). From the
> description it seems to be the right package:

That's the package I tried initially which failed all my (limited)
testing due to the hardware not being supported.

Mesa 23 is available as experimental, but I did not manage to install
that due to dependencies.

In the end, you'll have to test with your particular hardware, I think
-- I've not been able to find a reliable and realistic list of working
hard- and software combinations.



Current state, by the way, is that I'm fighting with boinc and systemd
in this environment. OpenCL reminds me of the (good|bad) days of
building my own kernel based upon incomplete information ;-)

Would be fun if I could sink a few days in it.

Cheers,

Arno

[Yvan Masson]

As I previously said, I know nothing about the subject so I might be out
of topic, but:
- in February 2022 someone wrote an interesting article, in French, of
the current (bad) state of OpenCL Linux software with AMD hardware
- this article points to some English documentation and scripts this
person maintain [2]

1.
https://linuxfr.org/news/opencl-sous-linux-l-etat-des-pilotes-amd-est-desormais-pire-que-ce-qu-il-etait-a-l-epoque-de-fglrx
2. https://gitlab.com/illwieckz/i-love-compute

Hope this helps.

[Sarunas Burdulis]

> Which leaves me with one question for this mail thread: Can anybody
> recommend a test program for OpenCL functionality?

There are tests from Khronos:
https://github.com/KhronosGroup/OpenCL-CTS

It's seems to be over 50 tests and then subtests. There are presets to
run different test suites from a Python script.

I ran only some random ones with ROCm and Mesa OpenCL, using Vega64 or
RX 5700 XT.

All tested fine with ROCm OpenCL. Mesa OpenCL failed some with Vega64
and then most tests with the RX 5700 XT...

This is Debian 12.

--
Sarunas Burdulis
Dartmouth Mathematics
math.dartmouth.edu/~sarunas

· https://useplaintext.email ·

[Arno Lehmann]

Thanks for pointing me there... I tried the suite myself, and got
results that at least confirmed some stuff works :-)

Cheers,