OMAPDRM eventual failure after

[Andre Renaud]

Hi,
I'm using the OmapDRM drivers (along with libDCE) to do accelerated
video playback. This is all working fine, but after a while I start
getting the following errors, and the whole system stops functioning:
[17126.949310] omapdrm omapdrm: could not remap: -12 (3)
[17126.954650] omapdrm omapdrm: could not swap e6b733c0 -> e6b73240
[17126.960968] omapdrm omapdrm: dispc_ovl_setup failed: -22

Has anyone seen this before? I had a vague feeling it might have to do
with memory fragmentation from my drm planes, but I'm not sure about
that.

On a possibly related note, I do see the following kernel output quite
regularly as well. This doesn't seem to cause any problems, although
the act of printing it to the console briefly slows things down a bit.
[17133.534881] ------------[ cut here ]------------
[17133.539733] WARNING: at drivers/staging/omapdrm/omap_irq.c:56
omap_irq_register+0x48/0x88()
[17133.548492] Modules linked in: omapdce(C) omap_rpmsg_resmgr
rpmsg_resmgr virtio_rpmsg_bus omap_remoteproc remoteproc virtio
virtio_ring
[17133.561340] [<c0016464>] (unwind_backtrace+0x0/0xec) from
[<c0587c74>] (dump_stack+0x20/0x24)
[17133.570281] [<c0587c74>] (dump_stack+0x20/0x24) from [<c003d488>]
(warn_slowpath_common+0x5c/0x74)
[17133.579711] [<c003d488>] (warn_slowpath_common+0x5c/0x74) from
[<c003d4cc>] (warn_slowpath_null+0x2c/0x34)
[17133.589843] [<c003d4cc>] (warn_slowpath_null+0x2c/0x34) from
[<c03fc3a4>] (omap_irq_register+0x48/0x88)
[17133.599670] [<c03fc3a4>] (omap_irq_register+0x48/0x88) from
[<c03fcf7c>] (apply_worker+0x104/0x13c)
[17133.609191] [<c03fcf7c>] (apply_worker+0x104/0x13c) from
[<c00592d4>] (process_one_work+0x264/0x418)
[17133.618774] [<c00592d4>] (process_one_work+0x264/0x418) from
[<c0059808>] (worker_thread+0x1b4/0x2cc)
[17133.628448] [<c0059808>] (worker_thread+0x1b4/0x2cc) from
[<c005db64>] (kthread+0x9c/0xa8)
[17133.637145] [<c005db64>] (kthread+0x9c/0xa8) from [<c000ef6c>]
(kernel_thread_exit+0x0/0x8)
[17133.645904] ---[ end trace da227214a8249215 ]---


Regards,
Andre

[Rob Clark]

There was a leak or two that was fixed recently.. IIRC it needs an
update in omapdce kernel module and in the ducati firmware.. not sure
if this is pushed yet

BR,
-R

[Andre Renaud]

Hi Rob,

On 27 September 2012 18:54, Rob Clark <robd...@gmail.com> wrote:
> There was a leak or two that was fixed recently.. IIRC it needs an
> update in omapdce kernel module and in the ducati firmware.. not sure
> if this is pushed yet

Do you know where this will be published through? I'm currently using
the dispc-hl-split branch of your
git://github.com/robclark/kernel-omap4.git tree, as this prevented the
flickering I was seeing on other kernels. Can you tell me if this work
has migrated through to any of the stock trees?

Thanks,
Andre

[Nicolas Dechesne]

On Thu, Sep 27, 2012 at 10:40 PM, Andre Renaud <an...@bluewatersys.com> wrote:
> Do you know where this will be published through? I'm currently using
> the dispc-hl-split branch of your
> git://github.com/robclark/kernel-omap4.git tree, as this prevented the
> flickering I was seeing on other kernels. Can you tell me if this work
> has migrated through to any of the stock trees?

otoh, that branch should be more recent
https://github.com/xboudet/linux/tree/ti-ubuntu-3.4-1487_dispc-hl-split

[Andre Renaud]

Hi Nicolas,

I jumped to that tree, and had some issues with the dce driver. Using
the firmware from tiomap4-syslink-mm-firmware v2.x.10.1, I get the
following error:
[ 629.490509] omap-rproc omap-rproc.1: received echo reply from ipu_c0
[ 634.490844] rproc remoteproc1: fatal error #17 detected in ipu_c0:
error type watchdog fired
[ 634.499725] rproc remoteproc1: trying to recover ipu_c0
[ 634.543121] omap_hwmod: ipu: failed to hardreset
[ 634.548187] rproc remoteproc1: stopped remote processor ipu_c0
[ 634.904205] rproc remoteproc1: powering up ipu_c0
[ 634.948181] rproc remoteproc1: Booting fw image
ducati-m3-core0.xem3, size 3595847
[ 634.956695] omap-iommu omap-iommu.1: ipu: version 2.1
[ 634.974334] rproc remoteproc1: dma_alloc_coherent failed: 102760448
[ 634.981109] rproc remoteproc1: Failed to process resources: -12
[ 635.006774] omap_hwmod: ipu: failed to hardreset
[ 635.012115] rproc remoteproc1: rproc_boot() failed -12
[ 635.017578] virtio_rpmsg_bus: probe of virtio17 failed with error -12
[ 635.024353] rproc remoteproc1: registered virtio17 (type 7)


If I switched to what I presume is the new firmware
ti-firmware-ipu-dce v1.6+120829+191835+git53cf578, I get the following
error:
[ 12.721282] rproc remoteproc1: powering up ipu_c0
[ 12.764373] rproc remoteproc1: Booting fw image
ducati-m3-core0.xem3, size 3595847
[ 12.772430] omap-iommu omap-iommu.1: ipu: version 2.1
[ 12.787353] rproc remoteproc1: dma_alloc_coherent failed: 102760448
[ 12.793975] rproc remoteproc1: Failed to process resources: -12
[ 12.819580] omap_hwmod: ipu: failed to hardreset
[ 12.827148] omap_hwmod: ipu: _wait_target_disable failed
[ 12.833221] rproc remoteproc1: rproc_boot() failed -12
[ 12.838653] virtio_rpmsg_bus: probe of virtio0 failed with error -12
[ 12.845367] rproc remoteproc1: registered virtio0 (type 7)

Any ideas what I might have missed?

Regards,
Andre

[Rob Clark]

On Fri, Sep 28, 2012 at 4:19 AM, Andre Renaud <an...@bluewatersys.com> wrote:
> Hi Nicolas,
>
> On 28 September 2012 09:17, Nicolas Dechesne <nde...@gmail.com> wrote:
>> On Thu, Sep 27, 2012 at 10:40 PM, Andre Renaud <an...@bluewatersys.com> wrote:
>>> Do you know where this will be published through? I'm currently using
>>> the dispc-hl-split branch of your
>>> git://github.com/robclark/kernel-omap4.git tree, as this prevented the
>>> flickering I was seeing on other kernels. Can you tell me if this work
>>> has migrated through to any of the stock trees?
>>
>> otoh, that branch should be more recent
>> https://github.com/xboudet/linux/tree/ti-ubuntu-3.4-1487_dispc-hl-split
>
> I jumped to that tree, and had some issues with the dce driver. Using
> the firmware from tiomap4-syslink-mm-firmware v2.x.10.1, I get the
> following error:

I guess there might be an updated firmware that should go along with
this kernel.. :-/

BR,
-R

[Andre Renaud]

Hi Rob,

>> I jumped to that tree, and had some issues with the dce driver. Using
>> the firmware from tiomap4-syslink-mm-firmware v2.x.10.1, I get the
>> following error:
>
> I guess there might be an updated firmware that should go along with
> this kernel.. :-/

Does anyone know when that firmware might be available?

Regards,
Andre

[Nicolas Dechesne]

we usually update the PPA the 1st week of every month... so later this
week is the current target!

>
> Regards,
> Andre

[Martin Ertsås]

Is the patches we need from here applied in the omapzoom kernel-ubuntu tree?

- Martin

[Nicolas Dechesne]

well.. not really ;-)

we actually have 2 variants of our ubuntu kernel... the one which is
on omapzoom is what we push in the PPA. rob has been working on a
redesign that impacts omapdrm/omapdss that is improving quite a few
things around overlay and compositing.. that's this 'dispc-hl-split'
branch. the problem with that branch is that it only supports hdmi for
now, not DSI, that's why it's not merged into TI ubuntu kernel.

rob and tomi (omadrm and omapdss maintainers) are working right now on
aligning on the proper design for these changes, and then they will be
merged into mainline kernel, and into TI ubuntu kernel.


>
> - Martin

[Andre Renaud]

Hi,

> we actually have 2 variants of our ubuntu kernel... the one which is
> on omapzoom is what we push in the PPA. rob has been working on a
> redesign that impacts omapdrm/omapdss that is improving quite a few
> things around overlay and compositing.. that's this 'dispc-hl-split'
> branch. the problem with that branch is that it only supports hdmi for
> now, not DSI, that's why it's not merged into TI ubuntu kernel.
>
> rob and tomi (omadrm and omapdss maintainers) are working right now on
> aligning on the proper design for these changes, and then they will be
> merged into mainline kernel, and into TI ubuntu kernel.

I'm trying to work out the correct combination of kernel & firmware to
get YUV & ARGB DRM layers going with no tearing effects. If I use the
dispc-hl-split branch from git://github.com/xboudet/linux.git, then I
don't seem to be able to use the newest dce firmware image, however if
I use the ti-ubuntu-3.4-1487.6 branch from
git://dev.omapzoom.org/pub/scm/integration/kernel-ubuntu.git, then I
get a horrible green flicker over the top of the YUV plane. I believe
this is a known issue that the dispc-hl-split fork is designed to
resolve.

So far I've had the most luck with Robs dispc-hl-split branch from
git://github.com/robclark/kernel-omap4.git (which is probably the same
as Xaviers). However I am experiencing some odd kernel
panics/warnings on that kernel that I was hoping would be resolved in
the ubuntu version.

Can anyone advise what the most appropriate setup should be?

Regards,
Andre

[Andre Renaud]

Hi,

> rob and tomi (omadrm and omapdss maintainers) are working right now on
> aligning on the proper design for these changes, and then they will be
> merged into mainline kernel, and into TI ubuntu kernel.

Can anyone offer any hints here as to what the best option for me
would be. I'm pretty sure I need the dispc-hl-split work, as it
prevents the on-screen flicker that I'm seeing without it, but the
memory leak that was recently fixed in the dce stuff (which I believe
is not yet fixed in the dispc-hl-split branch) is killing my
application. Can anyone point me to the commits that resolved the
memory leak issue?

Regards,
Andre

[Xavier Boudet TIF]

Hello Andre,

I would advise you to first dist-upgrade on TI PPA to get the last October update, please refer to following e-mail for instructions:
https://groups.google.com/forum/?fromgroups=#!topic/pandaboard/7DLabw7amBg

Then we have published a new "dspc-hl-split" kernel into the trunk PPA, please check kernel 3.4.0-1488.2:
https://launchpad.net/~tiomap-dev/+archive/omap-trunk/+sourcepub/2702738/+listing-archive-extra

NOTE: This kernel support only HDMI connector and do not support DVI.
NOTE2: corresponding source can be found in my github branch:
https://github.com/xboudet/linux/commits/ti-ubuntu-3.4-1488

Let me know if it fixes your issue.

Regards, Xavier Boudet

[Andre Renaud]

Hi Xavier,

We're not actually using the full Ubuntu distribution, just cherry
picking the pieces we want.

Unfortunately, using that kernel, we are still unable to load the
latest ducati firmware from
http://ppa.launchpad.net/tiomap-dev/omap-trunk/ubuntu/pool/main/t/ti-firmware-ipu-dce/ti-firmware-ipu-dce_1.6+121009+114852+git5c3f2bc.tar.gz

[ 20.136474] rproc remoteproc1: Booting fw image
ducati-m3-core0.xem3, size 3618829
[ 20.144531] omap-iommu omap-iommu.1: ipu: version 2.1
[ 20.159454] rproc remoteproc1: dma_alloc_coherent failed: 102760448
[ 20.166046] rproc remoteproc1: Failed to process resources: -12
[ 20.191619] omap_hwmod: ipu: failed to hardreset
[ 20.196990] rproc remoteproc1: rproc_boot() failed -12
[ 20.202423] virtio_rpmsg_bus: probe of virtio0 failed with error -12
[ 20.209106] rproc remoteproc1: registered virtio0 (type 7)

Any ideas about where we might be going wrong with this?

Regards,
Andre

[Rob Clark]

On Tue, Oct 9, 2012 at 4:31 PM, Andre Renaud <an...@bluewatersys.com> wrote:
> Hi Xavier,
>
> We're not actually using the full Ubuntu distribution, just cherry
> picking the pieces we want.
>
> Unfortunately, using that kernel, we are still unable to load the
> latest ducati firmware from
> http://ppa.launchpad.net/tiomap-dev/omap-trunk/ubuntu/pool/main/t/ti-firmware-ipu-dce/ti-firmware-ipu-dce_1.6+121009+114852+git5c3f2bc.tar.gz
>
> [ 20.136474] rproc remoteproc1: Booting fw image
> ducati-m3-core0.xem3, size 3618829
> [ 20.144531] omap-iommu omap-iommu.1: ipu: version 2.1
> [ 20.159454] rproc remoteproc1: dma_alloc_coherent failed: 102760448

hmm, if dma_alloc_coherent() is actually failing, that would point at
perhaps some kconfig issue? Maybe CMA not enabled, or rproc's CMA
pool is not big enough?

BR,
-R

[Andre Renaud]

>> [ 20.136474] rproc remoteproc1: Booting fw image
>> ducati-m3-core0.xem3, size 3618829
>> [ 20.144531] omap-iommu omap-iommu.1: ipu: version 2.1
>> [ 20.159454] rproc remoteproc1: dma_alloc_coherent failed: 102760448
>
>
> hmm, if dma_alloc_coherent() is actually failing, that would point at
> perhaps some kconfig issue? Maybe CMA not enabled, or rproc's CMA
> pool is not big enough?

At the moment it is set to 16MB, however the request there is for
~100MB. Is that reasonable?

Regards,
Andre

[Rob Clark]

Well, 16MiB is the default size for the global pool. But remoteproc
should have it's own pool. For a quick test you could just try to
make the global pool larger.

And yes, 100MiB is about what I'd expect... I wouldn't call it
*reasonable*, but that is OMX camera for you. (And I think we end up
using the same memory map for builds w/out OMX camera.)

BR,
-R

> Regards,
> Andre

[Andre Renaud]

Hi Rob,

>> At the moment it is set to 16MB, however the request there is for
>> ~100MB. Is that reasonable?
>
> Well, 16MiB is the default size for the global pool. But remoteproc
> should have it's own pool. For a quick test you could just try to
> make the global pool larger.
>
> And yes, 100MiB is about what I'd expect... I wouldn't call it
> *reasonable*, but that is OMX camera for you. (And I think we end up
> using the same memory map for builds w/out OMX camera.)

Thanks for that - the issue was resolved by bumping
CONFIG_OMAP4_IPU_CMA_SIZE to 0xA100000 (from 0x6500000). I'd
previously just been adjusting CONFIG_CMA_SIZE_MBYTES, which didn't
have any effect.

So has the newer firmware changed to add more features that consume
more memory here?

Regards,
Andre

[Rob Clark]

well, the memory increase is all about OMX camera.. which I don't
think is actually in the public firmware in the PPA. But I'm not 100%
sure about that offhand. I think the problem is that we don't have a
good way to have different memory maps for the internal firmware w/
OMX camera and the public firmware with just DCE on the same kernel.

BR,
-R

> Regards,
> Andre

[Nicolas Dechesne]

indeeed, this is the problem. our kernel is used for both public PPA,
but also for products/customers. So we need to be able to run DCE and
OMX camera, so we go with the worst case memory map... and no, OMX
camera is not pushed into the public PPA, we have it into another
firmware.

>
> BR,
> -R
>
>> Regards,
>> Andre

[Xavier Boudet]

Hi Andre,

I would say that you are missing Kernel config changes that corresponds to this new FW.

Can you clarify the kernel you are using?
Are you using source from: http://dev.omapzoom.org/?p=integration/kernel-ubuntu.git;a=summary
Are you using our config as reference: debian.ti-omap4/config/config.common.ubuntu

You shall have a look to the following patches, especially the kernel config one:
7c3423e UBUNTU: Config align CMA sizes for DSP and IPU (OMAP4 + OMAP5) with new SysBios'
64232eb remoteproc/omap: increase default CMA size for DSP by 2MB
f66f185 remoteproc/omap: increase default CMA size for OMAP4 IPU to 106MB
ceeb091 omap: remoteproc: adjust the CMA start addresses for OMAP4
be09d94 remoteproc/omap: correct the CMA sizes used for IPU & DSP

I guess you are using our code but your own config, so chcke commit: 7c3423e

Regards, Xavier Boudet

[Andre Renaud]

Hi Xavier,
I shuffled over to your kernel from github, and it does appear to be
behaving reasonably well. However I have struck a new issue - the DRM
subsystem seems to stall for about 100seconds 600 seconds after boot
up. During this period, all of my calls to drmHandleEvent
page_flip_handler return immediately (implying that vsync has just
occurred, but it hasn't), and nothing appears on screen. This rights
itself shortly later on, and then doesn't seem to reoccur. The timing
is remarkably consistent (100s of stall which starts 600s after
program startup).

Has anyone seen this issue before?

Regards,
Andre

On 8 October 2012 20:22, Xavier Boudet TIF <x-bo...@ti.com> wrote:

> Hello Andre,
>
> I would advise you to first dist-upgrade on TI PPA to get the last October
> update, please refer to following e-mail for instructions:
> https://groups.google.com/forum/?fromgroups=#!topic/pandaboard/7DLabw7amBg
>
> Then we have published a new "dspc-hl-split" kernel into the trunk PPA,
> please check kernel 3.4.0-1488.2:
> https://launchpad.net/~tiomap-dev/+archive/omap-trunk/+sourcepub/2702738/+listing-archive-extra
>
> NOTE: This kernel support only HDMI connector and do not support DVI.
> NOTE2: corresponding source can be found in my github branch:
> https://github.com/xboudet/linux/commits/ti-ubuntu-3.4-1488
>
> Let me know if it fixes your issue.
>
> Regards, Xavier Boudet

--
Bluewater Systems - An Aiotec Company

Andre Renaud
an...@bluewatersys.com 5 Amuri Park, 404 Barbadoes St
www.bluewatersys.com PO Box 13 889, Christchurch 8013
www.aiotec.co.nz New Zealand
Phone: +64 3 3779127 Freecall: Australia 1800 148 751
Fax: +64 3 3779135 USA 1800 261 2934

[Rob Clark]

hmm, it does sound like some roll-over issue.. IIRC, jiffies rolls
over some minutes (possibly 10 minutes) after boot.

Are you waiting for the fd to be signaled and then calling
drmHandleEvent()? Ie. select()/poll(), something along the lines of:
https://github.com/robclark/kmscube/blob/master/kmscube.c#L624

I'm curious if the event isn't happening at all, or if it is just a
problem reading back the event.

BR,
-R

[Andre Renaud]

Hi Rob,
Yes, I have the select/poll loop in there. I wondered about jiffies,
but it rolls at 5 minutes, not 10 minutes (which is what I'm seeing).
I've had a quick look through drivers/staging/omapdrm and
drivers/gpu/drm and can't see any obvious changes that would impact
this (I'm comparing it against your omap4 3.0.4 kernel).

Regards,
Andre

[Rob Clark]

just to rule something out, could you try:

echo 0 >/sys/devices/platform/omapdrm/graphics/fb0/blank

BR,
-R

[Andre Renaud]

Hi Rob,
That does fix it. So what is the story here - I never access the
/dev/fb device, only the /dev/card0 DRM device. Is this a
misconfiguration by my behalf, or a mistake in the kernel not
disabling the framebuffer when the DRM device is in use?

Regards,
Andre

[Rob Clark]

I'm not really sure.. I agree that if you aren't using the fbdev
device, that this shouldn't be effecting anything, so it seems like a
bug *somewhere*..

BR,
-R

[Andre Renaud]

Hi Rob,
As an intermediate fix I've just disabled CONSOLE_VT, I don't require
VTs for this application. I also found that adding consoleblank=0 to
the command line removes the problem. Obviously these are both just
work-arounds, but they are sufficient for me.

Regards,
Andre

[Rob Clark]

I actually suspect (fwiw), that the reason that x11 or wayland don't
have these issues is that they switch-vt.

BR,
-R

[Rob Clark]

fwiw, I brought this up on #dri-devel, and the conclusion was that it
isn't a "bug", but a "feature".. usespace should open the tty device,
and set the mode to KD_GRAPHICS

see, for example:
http://hub.opensolaris.org/bin/download/Project+vconsole/WebHome/vt.7i.txt

(it should be basically the same on linux)

or also:

http://www.linuxjournal.com/article/2783

it is a bit odd that the request to blank the display comes via the
legacy fbdev, but blanking the display is actually a function of the
VT.

BR,
-R