SGX on 4.4.*-ti-rt kernel

1,279 views
Skip to first unread message

matteo.fa...@sirius-es.it

unread,
Sep 26, 2016, 11:50:12 AM9/26/16
to BeagleBoard
Hi,

I try for the first time to enable SGX feauture on BBB using the kernel version 4.4.* to experiment and try to use it with weston.
I use jessie debian image (rcn) updated to kernel 4.4.16-ti-rt-r38
After the update, I install also sgx module:
$ apt-get install ti-sgx-ti335x-modules-4.4.16-ti-rt-r38

In this package there are:
pvrsrvkm.ko
bc_example.ko

I don't know if I need also omaplfb.ko.that I find in documentation.
Is there in any package? What packages I need to install for this kernel version?

Regards,
Matteo

Robert Nelson

unread,
Sep 26, 2016, 3:06:17 PM9/26/16
to Beagle Board, Matteo Facchinetti
So, you just need the userspace bits:

git clone -b ti-img-sgx/1.14.3699939
git://git.ti.com/graphics/omap5-sgx-ddk-um-linux.git --depth=1

cd omap5-sgx-ddk-um-linux/

sudo su
export DISCIMAGE=/
export TARGET_PRODUCT=ti335x
make install
exit

#fix: libgbm.so.2
sudo ln -s /usr/lib/arm-linux-gnueabihf/libgbm.so.1
/usr/lib/arm-linux-gnueabihf/libgbm.so.2

sudo mkdir -p /lib/modules/4.4.11-00332-gce54280-dirty/extra/
sudo ln -s /lib/modules/`uname -r`/extra/ti335x/pvrsrvkm.ko
/lib/modules/4.4.11-00332-gce54280-dirty/extra/pvrsrvkm.ko

debian@beaglebone:~$ sudo /usr/bin/pvrsrvctl --start
PVR:(Error): OpenServices: PVRDRMOpenRender failed [0, ]
PVR:(Error): PVRSRVInitSrvConnect: PVRSRVConnect failed [0, ]
PVR:(Error): SrvInit: PVRSRVInitSrvConnect failed (4) [0, ]
/usr/bin/pvrsrvctl: SrvInit failed (already initialized?)
(err=PVRSRV_ERROR_INIT_FAILURE)

Yeah, that's as far as i got... TI even got that far in our jessie image too...

Regards,


--
Robert Nelson
https://rcn-ee.com/
Message has been deleted
Message has been deleted
Message has been deleted
Message has been deleted
Message has been deleted
Message has been deleted
Message has been deleted

Matteo Facchinetti

unread,
Sep 28, 2016, 5:05:50 AM9/28/16
to BeagleBoard, matteo.fa...@sirius-es.it



Yeah, that's as far as i got... TI even got that far in our jessie image too...


 
Yeah, you are right..... SGX module is an headache.... inclusion of proprietary 
code. This is the main problem... :-(

...

Well,
with the same commands, have the same error...  good.

another little step:

The problem seems due to a wrong pvrsrvkm.ko in the package. I have opened a thread in TI forum to know what are the exactly module version to build.
[http://e2e.ti.com/support/arm/sitara_arm/f/791/p/544759/1987258#1987258]
And response is that we have to use the latest stack, integrated into the last Processor SDK: <Processor SDK>**/board-support/extra-drivers/** So in my host - download ti-processor-sdk-linux-rt-am335x-evm-03.00.00.04 and enter in the right path - copy board-support/extra-drivers/ti-sgx-ddk-km-1.14.3699939 directory in my BBB in my BBB: $ cd ti-sgx-ddk-km-1.14.3699939/eurasia_km/eurasiacon/build/linux2/omap_linux/ $ export DISCIMAGE=/ $ export TARGET_PRODUCT=ti335x $ export KERNELDIR=/usr/src/linux-headers-4.4.16-ti-rt-r38/ $ make So, copy by hand pvrsrvkm.ko in the right place: **/lib/modules/**<your kernel version>/extra/ti335x because "make install" doesn't work.. Arrrrrgggh Finally: $ sudo /usr/bin/pvrsrvctl --start module starts properly and kernel says: [ 208.971076] [drm] Initialized pvr 1.14.3699939 20110701 on minor 1 [ 208.974968] PVR_K: UM DDK-(3699939) and KM DDK-(3699939) match. [ OK ] But when I try to execute any example placed in omap5-sgx-ddk-um-linux/targetfs/ sound like libEGL warning: DRI3: xcb_connect failed libEGL warning: DRI2: xcb_connect failed libEGL warning: DRI2: xcb_connect failed Unable to initialise egl egl error 'EGL_NOT_INITIALIZED' (0x3001) Is possible that we have to update also userspace bin? Regards, Matteo

Matthijs van Duin

unread,
Oct 4, 2016, 3:24:46 PM10/4/16
to BeagleBoard, matteo.fa...@sirius-es.it, Matthijs van Duin
On Wednesday, 28 September 2016 11:05:50 UTC+2, Matteo Facchinetti wrote:
libEGL warning: DRI3: xcb_connect failed
libEGL warning: DRI2: xcb_connect failed
libEGL warning: DRI2: xcb_connect failed
Unable to initialise egl
    egl error 'EGL_NOT_INITIALIZED' (0x3001)

Sounds like it's not using the right libs, maybe check using ldd? It's trying to connect to X11, whereas these SGX drivers do not support X11, the only backends are Wayland, DRM, and GBM.

Matthijs

Matthijs van Duin

unread,
Oct 4, 2016, 4:34:44 PM10/4/16
to BeagleBoard, matteo.fa...@sirius-es.it, Matthijs van Duin
On Monday, 26 September 2016 21:06:17 UTC+2, RobertCNelson wrote:
sudo mkdir -p /lib/modules/4.4.11-00332-gce54280-dirty/extra/
sudo ln -s /lib/modules/`uname -r`/extra/ti335x/pvrsrvkm.ko
/lib/modules/4.4.11-00332-gce54280-dirty/extra/pvrsrvkm.ko

You can also just modprobe the module (don't forget to run depmod unless make install already does so) and then pass the --no-module argument to pvrsrvctl. 
 
debian@beaglebone:~$ sudo /usr/bin/pvrsrvctl --start
PVR:(Error): OpenServices: PVRDRMOpenRender failed [0, ]

I asked someone I know at imgtec; this means "it failed to open /dev/dri/renderD128 or whatever render node it was assigned". Does anything show up in /dev/dri at all? Anything interesting in kernel log?

I find it a bit disconcerting that the git repo doesn't work even though the one from the SDK ought to be a snapshot thereof afaik. (He agrees that Matteo's result with the driver from SDK looks like the kernel driver loaded okay, but the userspace binaries are trying to use mesa libs instead of the sgx ones.)

Matthijs

Robert Nelson

unread,
Oct 4, 2016, 5:18:48 PM10/4/16
to Beagle Board, Matteo Facchinetti, Matthijs van Duin
On Tue, Oct 4, 2016 at 3:34 PM, Matthijs van Duin
<matthij...@gmail.com> wrote:
> On Monday, 26 September 2016 21:06:17 UTC+2, RobertCNelson wrote:
>>
>> sudo mkdir -p /lib/modules/4.4.11-00332-gce54280-dirty/extra/
>> sudo ln -s /lib/modules/`uname -r`/extra/ti335x/pvrsrvkm.ko
>> /lib/modules/4.4.11-00332-gce54280-dirty/extra/pvrsrvkm.ko
>
>
> You can also just modprobe the module (don't forget to run depmod unless
> make install already does so) and then pass the --no-module argument to
> pvrsrvctl.

Oh i had forgotten about --no-module, I had found it once before in
ti's rc.pvr init script..

ti's rc.pvr:

https://gist.github.com/RobertCNelson/1a3a9ca14e537a706905027d6d9b60c0#file-rc-pvr


>
>>
>> debian@beaglebone:~$ sudo /usr/bin/pvrsrvctl --start
>> PVR:(Error): OpenServices: PVRDRMOpenRender failed [0, ]
>
>
> I asked someone I know at imgtec; this means "it failed to open
> /dev/dri/renderD128 or whatever render node it was assigned". Does anything
> show up in /dev/dri at all? Anything interesting in kernel log?

debian@beaglebone:~$ sudo modprobe pvrsrvkm

(nothing in dmesg)

debian@beaglebone:~$ lsmod | grep pvr
pvrsrvkm 379674 0
debian@beaglebone:~$ ls /dev/dri/*
/dev/dri/card0 /dev/dri/controlD64

debian@beaglebone:~$ sudo /usr/bin/pvrsrvctl --no-module --start
PVR:(Error): OpenServices: PVRDRMOpenRender failed [0, ]
PVR:(Error): PVRSRVInitSrvConnect: PVRSRVConnect failed [0, ]
PVR:(Error): SrvInit: PVRSRVInitSrvConnect failed (4) [0, ]
/usr/bin/pvrsrvctl: SrvInit failed (already initialized?)
(err=PVRSRV_ERROR_INIT_FAILURE)

X.org was running, so let's kill that and it's friends (x11-common removed)

no change...

> I find it a bit disconcerting that the git repo doesn't work even though the
> one from the SDK ought to be a snapshot thereof afaik. (He agrees that
> Matteo's result with the driver from SDK looks like the kernel driver loaded
> okay, but the userspace binaries are trying to use mesa libs instead of the
> sgx ones.)

Matthijs van Duin

unread,
Oct 4, 2016, 6:23:43 PM10/4/16
to Robert Nelson, Beagle Board, Matteo Facchinetti
On 4 October 2016 at 23:18, Robert Nelson <robert...@gmail.com> wrote:
debian@beaglebone:~$ sudo modprobe pvrsrvkm

(nothing in dmesg)

debian@beaglebone:~$ lsmod | grep pvr
pvrsrvkm              379674  0
debian@beaglebone:~$ ls /dev/dri/*
/dev/dri/card0  /dev/dri/controlD64

Those two nodes are from tilcdc obviously, so nothing in log and nothing in /dev/dri, that strongly suggests it's not matching against the device tree.
 
How was this module built?

Matthijs

Robert Nelson

unread,
Oct 4, 2016, 8:05:00 PM10/4/16
to Matthijs van Duin, Beagle Board, Matteo Facchinetti
https://github.com/rcn-ee/ti-sgx-omap5/blob/master/build-ti335x-4.4.x.sh

against a mirror of this tree/branch:

http://git.ti.com/gitweb/?p=graphics/omap5-sgx-ddk-linux.git;a=shortlog;h=refs/heads/ti-img-sgx/1.14.3699939/k4.4

make ARCH=arm CROSS_COMPILE= KERNELDIR=/build/buildd/linux-src clean
make ARCH=arm CROSS_COMPILE= KERNELDIR=/build/buildd/linux-src

/build/buildd/linux-src is where we just built the kernel, so
zImage/modules exist in that directory..

build log:

https://gist.github.com/RobertCNelson/1a3a9ca14e537a706905027d6d9b60c0#file-ti-sgx-ti335x-modules-4-4-23-ti-r51_1jessie_armhf-deb

William Hermans

unread,
Oct 4, 2016, 8:44:00 PM10/4/16
to beagl...@googlegroups.com
So libEGL is Mesa's Wayland extensions. Mesa is a 3d graphics library. Not sure how SGX even comes in at this context ?

Matthijs van Duin

unread,
Oct 4, 2016, 9:05:55 PM10/4/16
to Beagle Board
On 5 October 2016 at 02:43, William Hermans <yyr...@gmail.com> wrote:
So libEGL is Mesa's Wayland extensions.

 
Mesa is a 3d graphics library. Not sure how SGX even comes in at this context ?

Well it's not going to if you accidently use the wrong libEGL.so, but there's also one among the SGX userspace libs.

Matthijs

William Hermans

unread,
Oct 4, 2016, 9:37:25 PM10/4/16
to beagl...@googlegroups.com
 Yeah I know what EGL *is* but never used libEGL. Did a search wound up on Mesa's webpage where it said that. But now I can not find the same information. Which is funny because all I did was highlight "libEGL" from your post,right click, and google searched. Both times.

Matteo Facchinetti

unread,
Oct 17, 2016, 3:43:12 AM10/17/16
to BeagleBoard, matthij...@gmail.com, matteo.fa...@sirius-es.it

I belive to have discovered what is the problem:

in build-ti335x-4.4.x.sh script, when specify make_options, there isn't
"TARGET_PRODUCT=ti335x" option.

If not specify TARGET_PRODUCT, makefile compiles this module for the Jacinto SoC as default.

Matteo

Robert Nelson

unread,
Oct 17, 2016, 6:51:28 AM10/17/16
to Beagle Board

Nice find Matteo, I missed that the makefile grew a device option..

Regards,

Robert Nelson

unread,
Oct 17, 2016, 11:39:24 AM10/17/16
to Beagle Board, Matteo Facchinetti
That worked awesome, pvr's now talking:

[ 188.742066] [drm] Initialized pvr 1.14.3699939 20110701 on minor 1
[ 398.460105] PVR_K: UM DDK-(3699939) and KM DDK-(3699939) match. [ OK ]

tagging and pushing out:

4.4.24-ti-r56 with sgx module build fix..

#one line:
git clone -b ti-img-sgx/1.14.3699939
git://git.ti.com/graphics/omap5-sgx-ddk-um-linux.git --depth=1

cd omap5-sgx-ddk-um-linux/

sudo su
export DISCIMAGE=/
export TARGET_PRODUCT=ti335x
make install
exit

#fix: libgbm.so.2 (one line:)
sudo ln -s /usr/lib/arm-linux-gnueabihf/libgbm.so.1
/usr/lib/arm-linux-gnueabihf/libgbm.so.2

sudo modprobe pvrsrvkm
sudo /usr/bin/pvrsrvctl --start --no-module

Robert Nelson

unread,
Oct 17, 2016, 11:43:51 AM10/17/16
to Beagle Board, Matteo Facchinetti
The libraries are built for egl/wayland, so remember no x..

Charles Steinkuehler

unread,
Oct 17, 2016, 6:01:58 PM10/17/16
to beagl...@googlegroups.com
On 10/17/2016 10:42 AM, Robert Nelson wrote:
>
> The libraries are built for egl/wayland, so remember no x..

What's the status of wayland on the BBB with Debian Jessie?

...or is that more "build from source" fun, like the SGX drivers
themselves?

--
Charles Steinkuehler
cha...@steinkuehler.net

Robert Nelson

unread,
Oct 17, 2016, 6:06:03 PM10/17/16
to Beagle Board
On Mon, Oct 17, 2016 at 5:00 PM, Charles Steinkuehler
<cha...@steinkuehler.net> wrote:
> On 10/17/2016 10:42 AM, Robert Nelson wrote:
>>
>> The libraries are built for egl/wayland, so remember no x..
>
> What's the status of wayland on the BBB with Debian Jessie?
>
> ...or is that more "build from source" fun, like the SGX drivers
> themselves?

Well, my wayland script in jessie bombed:

https://github.com/RobertCNelson/boot-scripts/blob/master/tools/wm/weston-drm.sh

lots of tilcd noise over dmesg..

I'm pretty sure we will have to back-port wayland from stretch..

But this scares me:

http://git.ti.com/gitweb/?p=graphics/omap5-sgx-ddk-um-linux.git;a=commit;h=8d4ba04a2da59989b21aacca9defdf387e321695

the gcc 4.9 -> gcc 5.x was so much "FUN" in stretch over the summer, ^
might mean we have to just use stretch..

Matteo Facchinetti

unread,
Oct 19, 2016, 4:21:53 AM10/19/16
to BeagleBoard

I'm pretty sure we will have to back-port wayland from stretch..

But this scares me:

http://git.ti.com/gitweb/?p=graphics/omap5-sgx-ddk-um-linux.git;a=commit;h=8d4ba04a2da59989b21aacca9defdf387e321695

the gcc 4.9 -> gcc 5.x was so much "FUN" in stretch over the summer, ^
might mean we have to just use stretch..


I agree with you.... this commit could be determinant for choose a right strategy...this library now are built with gcc 5.x!!!

IMHO could be a good idea to work in stretch.
From debian.org site, on 2016-Nov will starts the freeze transition.
So packets could be considered quite "stable" and I think it's better to use a tested environment, for the wayland complexity.

Personally I don't like the back-port operation. The risk is to obtain a custom wayland version and discover new problems generated by backport, without debian support.

Should be fantastic to have a minimal stretch image with kernel 4.4 + SGX, only console runlevel 3, without X, as base to start to try wayland.

Other opinions are welcome.

Regards,
Matteo

Matteo Facchinetti

unread,
Oct 19, 2016, 10:14:56 AM10/19/16
to BeagleBoard
....... ops....  I use a wrong word...gramatical mistake.. sorry....
It would be fantastic to have a minimal stretch image....

Robert Nelson

unread,
Oct 19, 2016, 10:19:58 AM10/19/16
to Beagle Board, Matteo Facchinetti
Like:

http://elinux.org/Beagleboard:BeagleBoneBlack_Debian#microSD.2FStandalone:_.28testing.29_.28Bone.27s.29

PS, it's 4.9.0-rc1 based so make sure to run:

cd /opt/scripts/tools/
git pull
sudo ./update_kernel.sh --ti-channel --lts-4_4

(and the correctly built sgx module is now in the repo..)
Reply all
Reply to author
Forward
0 new messages