Raspberry Pi 4 sdl2/kmsdrm frame buffer transparency

[Alexander Lais]

Dear all,

I am updating my application that has been using OMXPlayer overlaid with a Kivy application (for some of the screens) to run on Raspberry Pi 4.
So far I have been using the egl_rpi backend on a specific layer that was ‘on top of’ the layer I use for the omxplayer video.

I’ve done a lot of research in the last weeks, coming to the conclusion that the dispmanx API is generally starting to be deprecated, which is fair enough.
There is a KMS / DRM backend for SDL2, which I’m using now that works to show the Kivy application and it works fine, but does not support transparent backgrounds.

I am currently using the Fake KMS mode that ultimately still uses the dispmanx in the background. This is for me a temporary solution that requires the least changes in my application at the moment and I’m considering a larger refactoring/change to not rely on dispmanx or the fake KMS more for the future.

In this mode I have found that the console frame buffer is on layer -127. I used `vcgencmd dispmanx_list`.
When I start my Kivy application, it takes over or replaces this layer -127 with an XRGB8888 buffer, meaning that this is RGB without per-pixel Alpha.
For my use case I need per pixel alpha, as I’m keeping the ‘middle’ of the screen transparent and add some decorations on top of the video that is shining through.

Going through the SDL2 code for the kmsdrm video mode, I saw that SDL sets this XRGB8888 pixel format, at least in SDL 2.0.10, which is the version recommended in the Kivy stable documentation.
Then I went on to check SDL 2.0.16 that was recently released and that sets the pixel for mat to ARGB8888, which should support alpha. It is likely that this was introduced in a version before .16 but I haven’d bisected all of them.

My latest step was to try to build SDL2 2.0.16 and run with that. Just rebuilding SDL2 (and _image) led to the app starting, but touch controls not working. It did also not solve my XRGB vs. ARGB issue.

My main question is now: How do I short-term get the Raspberry Pi to render a Kivy application with per-pixel alpha, and partially transparent sections?
Maybe there is some easier way that I’m just not seeing.


A later question is then: What is the intended way to do KMS / DRM based composition of different applications without using X or Wayland, and is there an intended way at all?


Thank you in advance,
Alex

[ElliotG]

I don't have a clue how to answer your question...  

Can you use a kivy video widget in place of the OMXPlayer?

[Alexander Lais]

I tried that but the frame rate on an RPi4 is 2-3fps with full HD video.

I will dig deeper to understand how KMS and DRM work and how SDL interacts with that to figure out how to resolve this. It seems that OMXPlayer will not be useful anymore with KMS/DRM. But there’s others such as VLC, gestreamer, etc.

It seems that what I wanted to do short term does not work and I will need to dig down into the long term question I had at the end of the original message.

Thanks for your support, Elliot.

On 16. Sep 2021, at 22:51, ElliotG <elli...@cox.net> wrote:

I don't have a clue how to answer your question...  

--
You received this message because you are subscribed to the Google Groups "Kivy users support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kivy-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kivy-users/78b7ead0-98f6-4baf-945e-ab4b392a1acfn%40googlegroups.com.

[Elliot Garbus]

You should try a different video backend: https://kivy.org/doc/stable/guide/environment.html?highlight=ffpyplayer

KIVY_VIDEO

Implementation to use for rendering video

Values: gstplayer, ffpyplayer, ffmpeg, null

To view this discussion on the web visit https://groups.google.com/d/msgid/kivy-users/074A5480-FDB3-44A0-BB02-A033E2C4440A%40me.com.

 

[Shashank Chintalagiri]

Alexander,

I find myself in pretty much the same boat as you. I would like to know if you found a reasonable way forward.

The following is the state of affairs as far my testing shows. Could you also confirm if this is along the lines of what you are seeing? 

PS : My apologies for the poorly structured notes, this is more stream of consciousness than a well thought though list. 

Drivers : 

 - buster on RPi4B uses vc4-fkms-v3d. vc4-kms-v3d may or may not work, I have not tried. 

 - bullseye on RPi4B uses vc4-kms-v3d. vc4-fkms-v3d does not boot to a virtual terminal (no screen visible).

Window Provider SDL2 :

 - SDL2 (by itself) (upto latest version, 2.0.22) and then kivy[full] compiles fine on both distributions. Setting the window backend to sdl2 works in that it displays the window. 

 - SDL2 on bullseye is noticeably laggy, irrespective of the presence of video. I'm seeing 5fps with simple animations. 

 - SDL2 also seems to compile OK on bullseye with --enable-arm-neon. This seems to kick things up to 12fps. Not sure if the cause and effect are closely related, though. I did a bunch of other things in between.

 - I have not yet tested SDL2 with --enable-arm-neon on buster. I hope to do so soon.

Video Provider ffmpeg:

 - This does not work at all. It seems to be unmaintained. 

Video Provider gstreamer:

 - This works reliably on both distributions. FPS of 1-2 (or even less) is what I get. Kivy's reported FPS (monitor module) is in the 20-40 range, but the video is obviously skipping frames.

Video Provider ffpyplayer:

 - This works on bullseye. FPS is again 5ish FPS, but I'm uncertain if it's a kivy / kivy installation / environment problem. It looks and feels exactly like the bad framerate during animations.

 - In order to make it work on bullseye, I installed additional packages ffmpeg (may not be needed?) and libavdevice-dev (needed). I only later noticed that libavdevice-dev seems to installing some version of libsdl2 from the apt repository. I currently suspect the use of this other version of libsdl is what is causing the sluggishness. It seems possible that manually compiling libavdevice against the correct  libsdl2 version may improve matters.

 - In order for the video to actually play and not just hang the whole system, I also had to install pulseaudio ( https://github.com/kivy/kivy/issues/6744 ). 

 - When libavdevice-dev is installed on buster, libsdl2 is broken. I suspect the buster libavdevice is built against a libsdl2 version < 2.0.10, which results in undefined symbol: SDL_RWread

 - As a result, ffpyplayer is not viable in buster without trying to compile libavdevice, which I have not yet worked up the courage to try. 

Kivy and Dispmanx : 

 - Upto RPi3, egl_rpi respected KIVY_BCM_DISPMANX_LAYER

 - RPi4 no longer supports egl_rpi. SDL2 does not seem to care about KIVY_BCM_DISPMANX_LAYER

 - Making the window background transparent has so far not worked (this code is suspect on my end, though, so I can't be sure it isn't something I broke). (Window.clearcolor = [0, 0, 0, 0])

Omxplayer

 - omxplayer on RPi4 Buster respects dispmanx layers, and works as intended.

 - omxplayer does not run on RPi4 Bullseye (vc4-kms-v3d)

 - When using RPi4 buster with SDL2, setting omxplayer -layer "-200" (something less than -127) does result in the omxplayer window not showing. I'm hoping it is because it is hidden behind it. 

FFplay

  - FFplay runs video perfectly on RPi4 Bullseye. 

  - It does not know about or support dispmanx layers, so moving it behind the kivy window using dispmanx is likely not viable. 

VLC

  - VLC does not seem to want to install and run without X. I suspect I will have nightmares of VLC's stderr outputs for some days. 

  - RPi4 Bullseye installing just vlc-bin does nothing useful. I ended up installing vlc, which pulled 1GB+ of packages with it. I later read that installing vlc-plugin-base may have been enough. 

  - When there's no X, vlc does not want to display any video with any combination of "-V aa",  "-V fb", "-I ncurses". The only luck I had was having the default result in an ascii art version of the video playing on my ssh terminal.

  - vlc is supposed to support some kind of MMAL output interface. I have no idea if it does, or how to make it do anything, but if I understand it correctly it should work on RPi4 Buster with vc4-fkms-v3d and do what we want.

During all of this, I have purposefully avoided checking if any audio is working, so can't comment on audio at all, other than that it worked fine on RPi3 and in the past on RPi4B Buster + omxplayer. 

 

 
Thanks and Regards

Shashank