[pandaboard] Overlays, Framebuffers and DRM

[Andre Renaud]

Hi,

Using the pandaboard, I'm trying to get a system set up where I can
combine a NV12/YUV overlay with an RGB framebuffer, without using X11.
However in order to use the newer DRM (which is a requirement to use
the ducati/h264 decode engine), the traditional FB_OMAP2 driver needs
to be disabled. Having done that, I'm not having difficulty attaching
fb0 to overlay0 - for instance running omapdrmtest, I'd like to be
able to put RGB data on top of the displayed video.

However, it would appear that /dev/fb0 is not attached to the display
once the DRM subsystem is initialised.

Is it possible to mix traditional framebuffer access with the new DRM
bits & pieces, or does the RGB data have to be accessed through libdrm
as well? I was previously under the impression that it was possible to
put /dev/fb0 into 32-bit ARGB mode, and use the alpha channel to
control how it was blended with the YUV channels, however I'm not not
sure if that is possible.

Regards,
Andre

[Rob Clark]

no, not really.. or well, it could probably be made to work in a hacky
way but it is not how kms/drm is intended to work. When a process
opens and is master on the drm device, it sets up it's own fb and
overlay (plane) layers. And then when the process exits, everything
is restored back to fbcon settings, so a process can't leave the
display in a bad state when it crashes or exits.

That said, you can have YUV or (A)RGB data on a kms plane. So you can
do what you want, but you should do it all through kms/drm APIs.
Which does have a few other advantages over fbdev (hotplug
notifications, more flexible way to handle flipping, etc).

BR,
-R

> Regards,
> Andre

[Andre Renaud]

Hi Rob,

> no, not really.. or well, it could probably be made to work in a hacky
> way but it is not how kms/drm is intended to work. When a process
> opens and is master on the drm device, it sets up it's own fb and
> overlay (plane) layers. And then when the process exits, everything
> is restored back to fbcon settings, so a process can't leave the
> display in a bad state when it crashes or exits.
>
> That said, you can have YUV or (A)RGB data on a kms plane. So you can
> do what you want, but you should do it all through kms/drm APIs.
> Which does have a few other advantages over fbdev (hotplug
> notifications, more flexible way to handle flipping, etc).

Is there any documentation on how the various bits & pieces of
libdrm/kms (connectors, encoders, crtcs etc...) all fit together? I'm
having a bit of grief getting my head around it all, and I can't seem
to find any documentation about from the test programs - modetest &
modeprint.

Regards,
Andre

[Rob Clark]

there is some relatively recent patch to update the kernel side documentation:

http://www.ideasonboard.org/media/drm/index.html

I'm not aware of something like this yet that is more oriented at the
userspace facing APIs, although the kernel side docs should explain a
bit planes/crtcs/encoders/connectors.

BR,
-R

> Regards,
> Andre

[Alison Chaiken]

Andre Renaud <an...@bluewatersys.com> asks:

> Is there any documentation on how the various bits & pieces of
> libdrm/kms (connectors, encoders, crtcs etc...) all fit together?

Here

http://blog.mecheye.net/2012/06/the-linux-graphics-stack/

is the best explanation ever, courtesy of Jasper St. Pierre and LWN.
See also amazing slidesets by Jim Huang of 0xLabs at

http://www.slideshare.net/jserv/design-and-concepts-of-android-graphics

for example. That one features a comparison of Android and X11
graphics stacks: excellent stuff.

--
Alison Chaiken
(650) 279-5600 (cell) {she-devel.com,
exerciseforthereader.org}
From what I've known of desire, I hold with those who favor fire.

[Andre Renaud]

Hi Rob,

> no, not really.. or well, it could probably be made to work in a hacky
> way but it is not how kms/drm is intended to work. When a process
> opens and is master on the drm device, it sets up it's own fb and
> overlay (plane) layers. And then when the process exits, everything
> is restored back to fbcon settings, so a process can't leave the
> display in a bad state when it crashes or exits.
>
> That said, you can have YUV or (A)RGB data on a kms plane. So you can
> do what you want, but you should do it all through kms/drm APIs.
> Which does have a few other advantages over fbdev (hotplug
> notifications, more flexible way to handle flipping, etc).

I've had a poke around, and essentially managed to get some version of
this working - enough that I can see how it will proceed. However I'm
now curious if I'm even going about this the correct way. I am trying
to develop an application that allows overlaying of transparent (PNG)
images, and antialiased text on top of high definition video. In the
past I've never done an embedded system using X, and I've always
assumed that it was too 'heavy-weight'. However with all of the recent
developments, perhaps I'm mistaken about that. Would you recommend
using libdrm directly, or skipping it, and using the higher level APIs
such as Cairo and gstreamer?

I like to understand all of the software elements of the systems we
produce, and with libdrm I think that is still going to be managable,
but I think once I go to gstreamer/Cairo/X etc... it will be too much
for me to have actually looked through the source for everything and
traced it through.

As a general question to the community - does anyone do any graphical
work outside of X windows now?

Regards,
Andre

[Rob Clark]

On Wed, Jul 25, 2012 at 11:22 PM, Andre Renaud <an...@bluewatersys.com> wrote:
> Hi Rob,
>> no, not really.. or well, it could probably be made to work in a hacky
>> way but it is not how kms/drm is intended to work. When a process
>> opens and is master on the drm device, it sets up it's own fb and
>> overlay (plane) layers. And then when the process exits, everything
>> is restored back to fbcon settings, so a process can't leave the
>> display in a bad state when it crashes or exits.
>>
>> That said, you can have YUV or (A)RGB data on a kms plane. So you can
>> do what you want, but you should do it all through kms/drm APIs.
>> Which does have a few other advantages over fbdev (hotplug
>> notifications, more flexible way to handle flipping, etc).
>
> I've had a poke around, and essentially managed to get some version of
> this working - enough that I can see how it will proceed. However I'm
> now curious if I'm even going about this the correct way. I am trying
> to develop an application that allows overlaying of transparent (PNG)
> images, and antialiased text on top of high definition video. In the
> past I've never done an embedded system using X, and I've always
> assumed that it was too 'heavy-weight'. However with all of the recent
> developments, perhaps I'm mistaken about that. Would you recommend
> using libdrm directly, or skipping it, and using the higher level APIs
> such as Cairo and gstreamer?

I would say if you want to bypass X, and you don't need sgx/opengles,
then you could use drm directly, in combination w/ gstreamer (kmssink)
and possibly something like cairo if you want.. IIRC libdrm modetest
app is using cairo to draw the test pattern.

> I like to understand all of the software elements of the systems we
> produce, and with libdrm I think that is still going to be managable,
> but I think once I go to gstreamer/Cairo/X etc... it will be too much
> for me to have actually looked through the source for everything and
> traced it through.
>
> As a general question to the community - does anyone do any graphical
> work outside of X windows now?

wayland :-P

(but we aren't quite there yet)

BR,
-R


> Regards,
> Andre

[Andre Renaud]

Hi Rob,

> no, not really.. or well, it could probably be made to work in a hacky
> way but it is not how kms/drm is intended to work. When a process
> opens and is master on the drm device, it sets up it's own fb and
> overlay (plane) layers. And then when the process exits, everything
> is restored back to fbcon settings, so a process can't leave the
> display in a bad state when it crashes or exits.
>
> That said, you can have YUV or (A)RGB data on a kms plane. So you can
> do what you want, but you should do it all through kms/drm APIs.
> Which does have a few other advantages over fbdev (hotplug
> notifications, more flexible way to handle flipping, etc).

This all essentially seems to work, however I'm having difficulty
getting the zorder values to 'stick' correctly. Given that the YUV
layer is technically below the ARGB layer, since the alpha value on
the RGB layer is typically set in such a way that the YUV layer shows
through, I was hoping to have overlay0 (ARGB) set to zorder 3, and
overlay3 (YUV) set to zorder 0. However overlay3 seems to be reset to
3 every time I call drmModeSetPlane, and then the kernel complains
that there are two enabled overlays with the same zorder.

I notice that there is a comment in the code for a TODO to add an
ioctl for this in the future. In the interim I have altered the
default settings in omap_plane.c:
526: omap_plane->info.zorder = 3 - ovl->id;
and am writing a 3 to /sys/devices/platform/omapdss/overlay0/zorder
after I've done my initial drmModeSetCrtc.

Any comments on how this should be done?

Regards,
Andre

[Rob Clark]

Yeah, there is a mechanism for drm crtc/plane properties (which we are
using for rotation).. the same should be used for configuring
z-order. Changing via sysfs isn't going to work (and actually the
sysfs interface should be removed or at least made read-only) since
omapdrm won't be aware of the change and will overwrite it.

BR,
-R

> Regards,
> Andre

[Andre Renaud]

Hi Rob,

> Yeah, there is a mechanism for drm crtc/plane properties (which we are
> using for rotation).. the same should be used for configuring
> z-order. Changing via sysfs isn't going to work (and actually the
> sysfs interface should be removed or at least made read-only) since
> omapdrm won't be aware of the change and will overwrite it.

When you say it should be used, do you mean I should be using the
drmModeGetProperty using the curren kernel, or do you mean that
omap_plane_set_property in omapdrm/omap_plane.c in the kernel should
be modified to add a zorder property (and then drmModeGetProperty
could be used)? From what I can see there is no support for any
property except rotation.

If a kernel patch is required, I'm happy to have a go at writing it, I
just want to be sure I'm working in the right area.

Regards,
Andre

[Rob Clark]

On Sun, Aug 5, 2012 at 9:02 PM, Andre Renaud <an...@bluewatersys.com> wrote:
> Hi Rob,
>
>> Yeah, there is a mechanism for drm crtc/plane properties (which we are
>> using for rotation).. the same should be used for configuring
>> z-order. Changing via sysfs isn't going to work (and actually the
>> sysfs interface should be removed or at least made read-only) since
>> omapdrm won't be aware of the change and will overwrite it.
>
> When you say it should be used, do you mean I should be using the
> drmModeGetProperty using the curren kernel, or do you mean that
> omap_plane_set_property in omapdrm/omap_plane.c in the kernel should
> be modified to add a zorder property (and then drmModeGetProperty
> could be used)? From what I can see there is no support for any
> property except rotation.

no, I haven't had a chance to implement it yet, but that would be the
proper way to expose this functionality to userspace

> If a kernel patch is required, I'm happy to have a go at writing it, I
> just want to be sure I'm working in the right area.

sure, patches welcome :-)

BR,
-R

> Regards,
> Andre

[Andre Renaud]

Hi Rob,

>> If a kernel patch is required, I'm happy to have a go at writing it, I
>> just want to be sure I'm working in the right area.
>
> sure, patches welcome :-)

What is the recommended patch submission process for this kind of
stuff? I'm only semi-competent with git, and not really familiar with
how to use the tools to automatically generate the submissions emails.
I've attached my preliminary version of the patch, which does seem to
allow me to control the zorder as I'd wanted without resorting to
sysfs.

Any comments on this?

Regards,
Andre

[Rob Clark]

Thanks. I'd suggest to use drm_property_create_range(), since we
don't really need an enum for a range of numerical values

And to be consistent, I'd use the DBG() macro instead of printk().

With those small tweaks, I would use 'git commit -s' (to attach
sign-off) and 'git format-patch' to generate a git patch (or git
send-email) to send it to dri-...@lists.freedesktop.org (and CC me
and gr...@kroah.com (GregKH)). See Documentation/SubmittingPatches

(or, well.. if you don't plan to send many kernel patches you can just
fwd the git formatted patch and I can send it.. but it is useful to
know how the kernel patch submission process works)

BR,
-R

> Regards,
> Andre

[Andre Renaud]

Hi Rob,

On 4 August 2012 01:24, Rob Clark <robd...@gmail.com> wrote:
> Yeah, there is a mechanism for drm crtc/plane properties (which we are
> using for rotation).. the same should be used for configuring
> z-order. Changing via sysfs isn't going to work (and actually the
> sysfs interface should be removed or at least made read-only) since
> omapdrm won't be aware of the change and will overwrite it.

I'm not having much luck applying the rotation property. Essentially
I'm calling drmModeObjectSetProperty(fd, 11, DRM_MODE_OBJECT_CRTC, 9,
0x2), however once I do this, drmModePageFlip starts returning
-ENOSPC. Do you know of any examples of how this should be used? I
presume this is because I need to swap the width/height at some point,
but I haven't quite worked out how to tie these together.

Regards,
Andre

[Rob Clark]

You need to allocated tiled buffers for rotation, so for 32bit
xRGB/ARGB, you would use OMAP_BO_TILED_32, ie:

struct omap_bo *bo = omap_bo_new_tiled(dev, 1920, 1080, OMAP_BO_TILED_32);
int handle = omap_bo_handle(bo); <-- gives you the handle to pass when
creating the fb

BR,
-R

> Regards,
> Andre

[Andre Renaud]

Hi Rob,

> You need to allocated tiled buffers for rotation, so for 32bit
> xRGB/ARGB, you would use OMAP_BO_TILED_32, ie:
>
> struct omap_bo *bo = omap_bo_new_tiled(dev, 1920, 1080, OMAP_BO_TILED_32);
> int handle = omap_bo_handle(bo); <-- gives you the handle to pass when
> creating the fb

That seems to work fine on the RGB surfaces, but when I'm using the
YUV planes, and then decoding H264 into them, it errors out in
VIDDEC3_process, even if I set outbufs->descs[x].memType =
XDM_MEMTYPE_TILED8/16, and the corresponding
bufSize.tileMem.width/height. Is it still correct to assign the handle
to outbufs->descs[0].buf when you're using tiled displays?

Regards,
Andre

[Rob Clark]

in theory you should just set MEMTYPE_BO for descs[0]/descs[1], and
that gets converted into the correct tiled typed on the kernel side.
It's possible there is something wrong in omapdce kernel part, I think
we have mostly not used 2d buffers yet w/ omapdce recently so it is
possible for there to be some bug.

BR,
-R

> Regards,
> Andre

[Andre Renaud]

Hi Rob,

> in theory you should just set MEMTYPE_BO for descs[0]/descs[1], and
> that gets converted into the correct tiled typed on the kernel side.
> It's possible there is something wrong in omapdce kernel part, I think
> we have mostly not used 2d buffers yet w/ omapdce recently so it is
> possible for there to be some bug.

On further review, this doesn't appear to really be related to H264
specifically - I am unable to get the YUV buffers to rotate at all,
nothing to do with DCE.

I assume that the YUV buffers must also be tiled to be rotated, so I'm
allocating them with omap_bo_new_tiled (there are two planes in the
YUV, Y & UV, so Y is widthxheightx8bpp, and UV is width * 0.5 x height
* 0.5 x 16bpp). However when I call drmModeSetPlane the kernel panics:

[ 61.050109] ------------[ cut here ]------------
[ 61.056152] WARNING: at arch/arm/mach-omap2/omap_l3_noc.c:117
l3_interrupt_handler+0x110/0x174()
[ 61.065399] L3 custom error: MASTER:DSS TARGET:DMM2
[ 61.070556] Modules linked in:
[ 61.073760] Backtrace:
[ 61.076354] [<c00126d4>] (dump_backtrace+0x0/0x118) from
[<c04b838c>] (dump_stack+0x20/0x24)
[ 61.085266] r6:c05e66b7 r5:00000075 r4:c06b9e40 r3:00000002
[ 61.091278] [<c04b836c>] (dump_stack+0x0/0x24) from [<c003c378>]
(warn_slowpath_common+0x5c/0x74)
[ 61.100616] [<c003c31c>] (warn_slowpath_common+0x0/0x74) from
[<c003c44c>] (warn_slowpath_fmt+0x40/0x48)
[ 61.110626] r8:c00315bc r7:000000c8 r6:00000180 r5:00080003 r4:f8000200
[ 61.117523] r3:00000009
[ 61.120300] [<c003c40c>] (warn_slowpath_fmt+0x0/0x48) from
[<c00316cc>] (l3_interrupt_handler+0x110/0x174)
[ 61.130493] r3:c06f2478 r2:c05e6707
[ 61.134277] [<c00315bc>] (l3_interrupt_handler+0x0/0x174) from
[<c0089134>] (handle_irq_event_percpu+0xbc/0x294)
[ 61.144989] r7:0000002a r6:e6900d80 r5:c06bc140 r4:e6900d80
[ 61.151000] [<c0089078>] (handle_irq_event_percpu+0x0/0x294) from
[<c0089358>] (handle_irq_event+0x4c/0x6c)
[ 61.161285] [<c008930c>] (handle_irq_event+0x0/0x6c) from
[<c008c224>] (handle_fasteoi_irq+0xd8/0x124)
[ 61.171081] r6:ffffffff r5:c06bc194 r4:c06bc140 r3:00000000
[ 61.177062] [<c008c14c>] (handle_fasteoi_irq+0x0/0x124) from
[<c0088a30>] (generic_handle_irq+0x30/0x40)
[ 61.187072] r5:00000000 r4:0000002a
[ 61.190856] [<c0088a00>] (generic_handle_irq+0x0/0x40) from
[<c000f184>] (handle_IRQ+0x88/0xc8)
[ 61.200042] r4:0000002a r3:0000021a
[ 61.203796] [<c000f0fc>] (handle_IRQ+0x0/0xc8) from [<c0008658>]
(gic_handle_irq+0x48/0x6c)
[ 61.212615] r5:c06b9f40 r4:fa240100
[ 61.216400] [<c0008610>] (gic_handle_irq+0x0/0x6c) from
[<c000dd80>] (__irq_svc+0x40/0x70)
[ 61.225128] Exception stack(0xc06b9f40 to 0xc06b9f88)
[ 61.225128] 9f40: 55d75555 fe404000 e5943010 fe404000 00000002
00000000 c07243c0 c0f7b1c0
[ 61.225158] 9f60: 8000406a 412fc09a 00000000 c06b9f94 c06b9f74
c06b9f88 c0024eb8 c000f4a8
[ 61.225158] 9f80: a0000113 ffffffff
[ 61.225158] r5:a0000113 r4:c000f4a8
[ 61.225189] [<c000f474>] (default_idle+0x0/0x3c) from [<c000f888>]
(cpu_idle+0xbc/0x120)
[ 61.225189] [<c000f7cc>] (cpu_idle+0x0/0x120) from [<c04a85ac>]
(rest_init+0x7c/0x94)
[ 61.225219] [<c04a8530>] (rest_init+0x0/0x94) from [<c0663838>]
(start_kernel+0x2c4/0x318)
[ 61.225219] r4:c06da560 r3:c0698d98
[ 61.225219] [<c0663574>] (start_kernel+0x0/0x318) from [<80008044>]
(0x80008044)
[ 61.225250] ---[ end trace 1b75b31a2719ed1f ]---

Has anyone had any luck using tiled YUV buffers?

Regards,
Andre

[Rob Clark]

I suspect setting the DOUBLESTRIDE bit in DISPC_VIDp_ATTRIBUTES is
missing.. this is needed for NV12 in particular.

BR,
-R

> Regards,
> Andre

[Andre Renaud]

Hi Rob,

> I suspect setting the DOUBLESTRIDE bit in DISPC_VIDp_ATTRIBUTES is
> missing.. this is needed for NV12 in particular.

Thanks for that hint Rob - certainly adding in that flag has resolved
the kernel panic, although I'm still having difficulty getting my
software going, but I assume it is just a mismatch about the sizes in
different places (sometimes you want the non-rotated width/height,
other times you want the rotated versions). I get a lot of the
following errors:
[ 15.877136] omapdss OVERLAY error: overlay 3 vertically not inside
the display area (0 + 1920 >= 1080)
[ 15.887817] omapdss APPLY error: failed to enable overlay 3:
check_settings failed

If you've got any comments on exactly how the rotation should be done,
I'd be appreciative, but with any luck I can work it out. So far, it
only seems to be working with 180 degree rotations (and 0) on tiled
buffers (not RAW), and I haven't worked out how to get the H264 engine
to work with the tiled buffers yet.

Here is a rough version of the change I made (please excuse Gmail's
messing up of the tabs/spaces).

diff --git a/drivers/video/omap2/dss/dispc.c b/drivers/video/omap2/dss/dispc.c
index c56a51a..682870b 100644
--- a/drivers/video/omap2/dss/dispc.c
+++ b/drivers/video/omap2/dss/dispc.c
@@ -971,6 +971,11 @@ static void dispc_ovl_set_vid_color_conv(enum
omap_plane plane, b
dispc_write_reg(DISPC_OVL_ATTRIBUTES(plane), val);
}

+static void dispc_ovl_set_doublestride(enum omap_plane plane, bool enable)
+{
+ REG_FLD_MOD(DISPC_OVL_ATTRIBUTES(plane), enable ? 1 : 0, 22, 22);
+}
+
static void dispc_ovl_enable_replication(enum omap_plane plane, bool enable)
{
static const unsigned shifts[] = { 5, 10, 10, 10 };
@@ -2240,6 +2245,8 @@ int dispc_ovl_setup(enum omap_plane plane,
struct omap_overlay_i

dispc_ovl_set_rotation_attrs(plane, oi->rotation, oi->mirror,
oi->color_mode);
+ dispc_ovl_set_doublestride(plane,
+ oi->color_mode == OMAP_DSS_COLOR_NV12);

dispc_ovl_set_zorder(plane, oi->zorder);
dispc_ovl_set_pre_mult_alpha(plane, oi->pre_mult_alpha);

Regards,
Andre

--
Bluewater Systems - An Aiotec Company

Andre Renaud
an...@bluewatersys.com 5 Amuri Park, 404 Barbadoes St
www.bluewatersys.com PO Box 13 889, Christchurch 8013
www.aiotec.co.nz New Zealand
Phone: +64 3 3779127 Freecall: Australia 1800 148 751
Fax: +64 3 3779135 USA 1800 261 2934

[Andre Renaud]

> Here is a rough version of the change I made (please excuse Gmail's
> messing up of the tabs/spaces).

On further review, this patch appears to mess up the UV plane, so this
doesn't seem to resolve the problem.

Regards,
Andre