Reverse engineering SGX 540 userspace driver

[Lucas Fryzek]

Hello,

I have a creator ci20 board sitting around that I haven't been using
very much and I know that the PowerVR GPUs have not had a lot of
progress on reverse engineering the driver for the GPU.

Does anyone know of any ongoing efforts to reverse engineer the SGX 540
driver? Or does anyone know of any documentation generated by previous
reverse engineering efforts? I'd love to contribute a helping hand in
reverse engineering how this GPU works.

Regards,
Lucas Fryzek

[H. Nikolaus Schaller]

Hi Lukas,

Yes, any help is very welcome!

I am aware of some projects:

a) OpenPVRSGX tries to include a DDK 1.14 or DDK 1.17 kernel driver - there is one version compatible to the jz4780
It explicitly excludes reverse engineering and focusses on kernel driver (which always was open source)
Home page: https://github.com/openpvrsgx-devgroup/linux_openpvrsgx
Mailing list: https://lists.goldelico.com/pipermail/openpvrsgx-devgroup
https://github.com/openpvrsgx-devgroup/linux_openpvrsgx/wiki/DDK-versions

b) LetuxOS works on making the CI20 basics useable (currently we try to upstream the HDMI driver)
The status is that we can compile the kernel driver in different DDK releases but have no matching
user-space code to test anything.
I thought about using the mipsel version of qemu-arm-static to run the arm pvrsrvctl tool to
at least download firmware but did not find time to really try it.

c) there is a n900 originated project to improve user-space code
and if I understand correctly they have build a compatible user-space
https://lists.goldelico.com/pipermail/openpvrsgx-devgroup/2021-October/000472.html
https://github.com/freemangordon/mesa/tree/mesa-pvr-ti

d) next, there was an FSF supported project with a collection of information but the original server has been lost
https://www.phoronix.com/scan.php?page=news_item&px=MTE0MDc
http://powervr.gnu.org.ve - Unfortunately the wayback machine has only archived 302 errors
Maybe there exists some other archive.
Back then there was material about partially RE instruction set etc.

e) finally there was a leak several years ago when some Android device vendor unintentionally published the
source code of the user-space code and shader compilers on github. This was retracted. There was some heavy
discussion and it was considered unethical by the community to make any use of it (even looking inside and
writing down observations).

>
> Regards,
> Lucas Fryzek

Hope this helps to get new ideas and started!

BR,
Nikolaus

[Lucas Fryzek]

(Re-sending as I forgot to reply-all to the whole mailing list)
Thanks for the quick response Nikolaus!

> a) OpenPVRSGX tries to include a DDK 1.14 or DDK 1.17 kernel driver -
> there is one version compatible to the jz4780
> It explicitly excludes reverse engineering and focusses on kernel
> driver (which always was open source)
> Home page: https://github.com/openpvrsgx-devgroup/linux_openpvrsgx
> Mailing list:
> https://lists.goldelico.com/pipermail/openpvrsgx-devgroup
>
> https://github.com/openpvrsgx-devgroup/linux_openpvrsgx/wiki/DDK-versions
>

Given that groups goal is to focus on the already open source kernel
driver, do you know if anyone in this group would have been involved in
any reverse engineering efforts?

> b) LetuxOS works on making the CI20 basics useable (currently we try
> to upstream the HDMI driver)
> The status is that we can compile the kernel driver in different
> DDK releases but have no matching
> user-space code to test anything.
> I thought about using the mipsel version of qemu-arm-static to run
> the arm pvrsrvctl tool to
> at least download firmware but did not find time to really try it.

Does the this updated kernel mode driver work with the latest userspace
DDK from imgtec (1.14.3759903)? I'm trying to get a setup right now
where I can compile a kernel mode driver and make modifications to get
a better idea on how the GPU is being controlled. If I can use a more
modern kernel and linux userland that would be very nice.

> c) there is a n900 originated project to improve user-space code
> and if I understand correctly they have build a compatible
> user-space
>
> https://lists.goldelico.com/pipermail/openpvrsgx-devgroup/2021-October/000472.html
> https://github.com/freemangordon/mesa/tree/mesa-pvr-ti
>

Thanks for that mailing list link! I'll try reaching out Ivaylo on that
mailing list and see what exactly the nature of their work is. If I
understand their email correctly it seems like they already managed to
get a RE ES2 driver working.

> d) next, there was an FSF supported project with a collection of
> information but the original server has been lost
> https://www.phoronix.com/scan.php?page=news_item&px=MTE0MDc
> http://powervr.gnu.org.ve - Unfortunately the wayback machine has
> only archived 302 errors
> Maybe there exists some other archive.
> Back then there was material about partially RE instruction set
> etc.

I took a look at wayback machine and I found a version of the project
properly recorded, and I found the following links.

https://web.archive.org/web/20130225101912/http://powervr.gnu.org.ve/doku.php?id=opcodes
https://web.archive.org/web/20151004182955/http://powervr.gnu.org.ve/doku.php?id=instructionencoding
http://lkcl.net/powervr/sgx/

[H. Nikolaus Schaller]

Hi Lukas,

> Am 10.11.2021 um 02:51 schrieb Lucas Fryzek <lucas....@hazeco.xyz>:
>
> (Re-sending as I forgot to reply-all to the whole mailing list)
> Thanks for the quick response Nikolaus!
>
>> a) OpenPVRSGX tries to include a DDK 1.14 or DDK 1.17 kernel driver - there is one version compatible to the jz4780
>> It explicitly excludes reverse engineering and focusses on kernel driver (which always was open source)
>> Home page: https://github.com/openpvrsgx-devgroup/linux_openpvrsgx
>> Mailing list: https://lists.goldelico.com/pipermail/openpvrsgx-devgroup
>> https://github.com/openpvrsgx-devgroup/linux_openpvrsgx/wiki/DDK-versions
> Given that groups goal is to focus on the already open source kernel driver, do you know if anyone in this group would have been involved in any reverse engineering efforts?
>
>
>> b) LetuxOS works on making the CI20 basics useable (currently we try to upstream the HDMI driver)
>> The status is that we can compile the kernel driver in different DDK releases but have no matching
>> user-space code to test anything.
>> I thought about using the mipsel version of qemu-arm-static to run the arm pvrsrvctl tool to
>> at least download firmware but did not find time to really try it.
> Does the this updated kernel mode driver work with the latest userspace DDK from imgtec (1.14.3759903)?

It *could* work but was never tested. We have the matching DDK release version in the tree. Just choose CONFIG_PVRSGX_1_14_3759903=y
It may be that the branch is missing some fixes that have been applied to the versions CONFIG_PVRSGX_1_14_3699939 and CONFIG_PVRSGX_1_17_4948957 which are more or less well running on OMAP3/4/5.

Do you have the user space for 1.14.3759903? I am not sure but it may be stored in the factory flash image.

> I'm trying to get a setup right now where I can compile a kernel mode driver and make modifications to get a better idea on how the GPU is being controlled. If I can use a more modern kernel and linux userland that would be very nice.

Just get the letux tree: https://github.com/goldelico/letux-kernel/tree/letux-5.15

It includes everything incl. HDMI driver for the CI20 (use letux_defconfig) and compiles the SGX driver that is configured. But the default is CONFIG_PVRSGX_1_17_4948957=y so there may be surprises when switching to CONFIG_PVRSGX_1_14_3759903 we can discuss how to fix.

>
>
>> c) there is a n900 originated project to improve user-space code
>> and if I understand correctly they have build a compatible user-space
>> https://lists.goldelico.com/pipermail/openpvrsgx-devgroup/2021-October/000472.html
>> https://github.com/freemangordon/mesa/tree/mesa-pvr-ti
> Thanks for that mailing list link! I'll try reaching out Ivaylo on that mailing list and see what exactly the nature of their work is. If I understand their email correctly it seems like they already managed to get a RE ES2 driver working.
>
>> d) next, there was an FSF supported project with a collection of information but the original server has been lost
>> https://www.phoronix.com/scan.php?page=news_item&px=MTE0MDc
>> http://powervr.gnu.org.ve - Unfortunately the wayback machine has only archived 302 errors
>> Maybe there exists some other archive.
>> Back then there was material about partially RE instruction set etc.
>
> I took a look at wayback machine and I found a version of the project properly recorded, and I found the following links.
>
> https://web.archive.org/web/20130225101912/http://powervr.gnu.org.ve/doku.php?id=opcodes
> https://web.archive.org/web/20151004182955/http://powervr.gnu.org.ve/doku.php?id=instructionencoding
> http://lkcl.net/powervr/sgx/

Ah Luke has had materials. I forgot about.

BR,
Nikolaus

[Lucas Fryzek]

Hello Nikolaus,

>
> It *could* work but was never tested. We have the matching DDK
> release version in the tree. Just choose CONFIG_PVRSGX_1_14_3759903=y
> It may be that the branch is missing some fixes that have been
> applied to the versions CONFIG_PVRSGX_1_14_3699939 and
> CONFIG_PVRSGX_1_17_4948957 which are more or less well running on
> OMAP3/4/5.
>
> Do you have the user space for 1.14.3759903? I am not sure but it may
> be stored in the factory flash image.
>
>> I'm trying to get a setup right now where I can compile a kernel
>> mode driver and make modifications to get a better idea on how the
>> GPU is being controlled. If I can use a more modern kernel and linux
>> userland that would be very nice.
>
> Just get the letux tree:
> https://github.com/goldelico/letux-kernel/tree/letux-5.15
>
> It includes everything incl. HDMI driver for the CI20 (use
> letux_defconfig) and compiles the SGX driver that is configured. But
> the default is CONFIG_PVRSGX_1_17_4948957=y so there may be surprises
> when switching to CONFIG_PVRSGX_1_14_3759903 we can discuss how to
> fix.

I believe my board is currently flashed with the `Debian 8 2015-09-09`
image from this webpage https://elinux.org/CI20_Distros.
I took a look through the filesystem and I cannot find the DDK as a zip
package, but all of the PVR libraries seem to be versioned as
`1.14.544606452`. The reason I referred to `1.14.3759903` before was
this is the version listed on this web page
https://elinux.org/CI20-SGX_kernel_module.


I'm not sure what the latest available kernel module + userland is for
the CI20. I did find this git repo
https://github.com/MIPS/CI20_PVR-prebuilts, that seems to have userland
libraries for version `1.15.4568187`. But I'm not if there is a
matching open source kernel driver for this repo.


I am currently trying to get a setup where I can build the open PVR
kernel module and have it work with the userspace. I've been working.
Building my own kernel with the old debian images seems to break DRM,
so I suspect there is so version mismatching going on. I'll take a look
at the kernel in the letux project instead as this looks like a more
productive use of my time. I'll also take a look at some of the older
debian images for the CI20 to see if the `1.14.3759903` to attempt to
use with the letux kernel.

Thanks for the help on first steps!

>

[Lucas Fryzek]

So I managed to get the letux kernel building and running on my CI20.
Had a bit of trouble there where the prebuilt kernels don't see to run
properly on the CI20, but once I grabbed the latest kernel from github
and built that everything seemed to work fine. I managed to find the
`1.14.3759903` userland at this link
https://ftp.radix.pro/3pp/Imagination/ci20/

I changed the kernel config to build the kernel module version
`1.14.3759903`, there were several errors but I just looked at the
`1.17` version and copied the fixed from there. Unfortunately the
userland does not seem to be happy, I tried testing a few of the
applications and none of them seem to work. I ran `strace` on the
binaries and it looks like they call `DRM_IOCTL_VERSION`, are unhappy
with results and then error out saying they can't find a PVR GPU.

I think to do any effective reverse engineering of the GPU I'll need a
working userland, so it looks like I have a bit of work there :P The
DRM version error gives a path to start looking, so I'll try messing
with that and see where I can go from there.

[H. Nikolaus Schaller]

That is interesting that there are even more intermediate DDK relases...

I think the format is
1.$RELEASE.$VERSION

where $VERSION is just a sequential number of their version management system. I.e. it increments only and independently of $RELEASE.
$RELEASE seems to be sort of milestone when they have tagged a specific $VERSION as some significant change.

But that said I have never seen source code for 1.14.544606452. And that may even break my assumption of sequential $VERSION numbers...
Maybe this has the scheme:
1.$RELEASE.$PATCH$VERSION i.e.

RELEASE=14
VERSION=4606452
PATCH=54

Then, the VERSION would fall between the OMAP 1.14.3699939 and the 1.17.4948957

> --
> You received this message because you are subscribed to the Google Groups "MIPS Creator CI20 Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mips-creator-ci2...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/mips-creator-ci20-dev/CCIF2R.T1CSDFANBVL12%40hazeco.xyz.

[H. Nikolaus Schaller]

HI Lucas,

Am 12.11.2021 um 05:18 schrieb Lucas Fryzek <lucas....@hazeco.xyz>:

Hello Nikolaus,

It *could* work but was never tested. We have the matching DDK release version in the tree. Just choose CONFIG_PVRSGX_1_14_3759903=y
It may be that the branch is missing some fixes that have been applied to the versions CONFIG_PVRSGX_1_14_3699939 and CONFIG_PVRSGX_1_17_4948957 which are more or less well running on OMAP3/4/5.
Do you have the user space for 1.14.3759903? I am not sure but it may be stored in the factory flash image.

I'm trying to get a setup right now where I can compile a kernel mode driver and make modifications to get a better idea on how the GPU is being controlled. If I can use a more modern kernel and linux userland that would be very nice.

Just get the letux tree: https://github.com/goldelico/letux-kernel/tree/letux-5.15
It includes everything incl. HDMI driver for the CI20 (use letux_defconfig) and compiles the SGX driver that is configured. But the default is CONFIG_PVRSGX_1_17_4948957=y so there may be surprises when switching to CONFIG_PVRSGX_1_14_3759903 we can discuss how to fix.

I believe my board is currently flashed with the `Debian 8 2015-09-09` image from this webpage https://elinux.org/CI20_Distros.
I took a look through the filesystem and I cannot find the DDK as a zip package, but all of the PVR libraries seem to be versioned as `1.14.544606452`. The reason I referred to `1.14.3759903` before was this is the version listed on this web page https://elinux.org/CI20-SGX_kernel_module.
I'm not sure what the latest available kernel module + userland is for the CI20. I did find this git repo https://github.com/MIPS/CI20_PVR-prebuilts, that seems to have userland libraries for version `1.15.4568187`. But I'm not if there is a matching open source kernel driver for this repo.
I am currently trying to get a setup where I can build the open PVR kernel module and have it work with the userspace. I've been working. Building my own kernel with the old debian images seems to break DRM, so I suspect there is so version mismatching going on. I'll take a look at the kernel in the letux project instead as this looks like a more productive use of my time. I'll also take a look at some of the older debian images for the CI20 to see if the `1.14.3759903` to attempt to use with the letux kernel.

So I managed to get the letux kernel building and running on my CI20.

That is great!

Had a bit of trouble there where the prebuilt kernels don't see to run properly on the CI20, but once I grabbed the latest kernel from github and built that everything seemed to work fine. I managed to find the `1.14.3759903` userland at this link https://ftp.radix.pro/3pp/Imagination/ci20/

Cool! So I should fetch that for my experiments or at least to support you with yours...

I changed the kernel config to build the kernel module version `1.14.3759903`, there were several errors but I just looked at the `1.17` version and copied the fixed from there.

Maybe, can you git-format-patch and submit on this list so that I can add them to the branch on github?

Unfortunately the userland does not seem to be happy, I tried testing a few of the applications and none of them seem to work. I ran `strace` on the binaries and it looks like they call `DRM_IOCTL_VERSION`, are unhappy with results and then error out saying they can't find a PVR GPU.

Yes... IMG did everything to make sure that kernel driver and user-space are compatible. So user-space cross-checks with the kernel drivers for two things I am aware of:

a) the version number

b) some compile flags and options

For a) I had thought about adding a mechanism that pvrdrv can report an arbitrary version number (maybe through some /sysfs entry).

For compile flags and options it was a little experimental until there were no more complaints.

AFAIR, the 1.14.3699939 user space for omap3 could be run in debug mode which told which flags were missing.

I guess this allows IMG to change data structures, features etc. of both UM and KM in parallel and avoid efforts to remain backwards compatible.

This is contrary to the idea of stable interfaces we usually have...

I think to do any effective reverse engineering of the GPU I'll need a working userland, so it looks like I have a bit of work there :P The DRM version error gives a path to start looking, so I'll try messing with that and see where I can go from there.

Yes, that is IMHO the right thing to start with. It gave me some insights how the whole pieces fit togehter (but I forgot most of them over the years :).

BR and thanks for sharing any thoughts and findings,

Nikolaus

[H. Nikolaus Schaller]

Am 12.11.2021 um 10:58 schrieb H. Nikolaus Schaller <h...@goldelico.com>:

So I managed to get the letux kernel building and running on my CI20. 

That is great!

Ah, one more note: it is not clear if the kernel driver does setup of clocks properly.

So the driver may load but either bail out or fail to download the firmware at some point.