LTDC display usage on the Stm32f429i board

[Nurettin SENYER]

Hi,

I run successfully 7 inch display (SATOZ) with the LTDC interface on the Stm32f429i board. But i cannot get result with same configuration on NuttX. How/where to start for the LTDC-external display on the stm32f429i board with NuttX? Can you share any guide/linkt/tutorial?

Our configuration are

- Board: Stm32f429i

- Interface: LTDC

- TFT: SAT070CP40D21Y0351071KTP

- GUI: LittlevGL v5 (we tested with custome LittlevGL configuration. It is very impressive for us)

Thanks for your time and interest,

https://github.com/seyyah

[Alan Carvalho de Assis]

Hi Nurettin,

I don't know details about LTDC hardware of STM32F429, but I think you
will need to modify the file
nuttx/configs/stm32f429i-disco/src/stm32_lcd.c to work with your LCD.

This contribution was done by Marco Krahl, he is who knows more about it.

I used the LTDC in the past when I ported FGL to NuttX and get the 3D
Cube demo working on it: https://www.youtube.com/watch?v=XOdfvi7cgLc
but I didn't modify the LCD driver.

BR,

Alan

> --
> You received this message because you are subscribed to the Google Groups
> "NuttX" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nuttx+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[Nurettin SENYER]

8 Mayıs 2018 Salı 16:17:35 UTC+3 tarihinde Alan Carvalho de Assis yazdı:

Hi Nurettin,

Hi Alan,

 

I don't know details about LTDC hardware of STM32F429, but I think you
will need to modify the file
nuttx/configs/stm32f429i-disco/src/stm32_lcd.c to work with your LCD.

I tried to but i could not work.

 

This contribution was done by Marco Krahl, he is who knows more about it.

I will contact to "Marco Krahl <ocram...@gmail.com>"

 

I used the LTDC in the past when I ported FGL to NuttX and get the 3D
Cube demo working on it: https://www.youtube.com/watch?v=XOdfvi7cgLc
but I didn't modify the LCD driver.

It could work with LittlevGL + Frame Buffer + LTDC on the Stm32f429i-disco. We changed a litlle confiuration on the initialization code. On the NuttX we tried a lot of things but we could not.

 

BR,

Thanks a lot for your quick response,

Nurettin

[Nurettin SENYER]

I started with simple step,

```sh

make distclean

./tools/configure.sh stm32f429i-disco/fb

vim configs/stm32f429i-disco/include/board.h

# gist: https://gist.github.com/seyyah/eb27986744547a98b6b5c0229285c85b#file-board-h

```

Gist: https://gist.github.com/seyyah/eb27986744547a98b6b5c0229285c85b#file-config

Make and write,

```sh

make

st-flash write nuttx.bin 0x8000000

```

We receive the below log,

```sh

up_hardfault: PANIC!!! Hard fault: 40000000

up_assert: Assertion failed at file:armv7-m/up_hardfault.c line: 171

up_dumpstate: sp:         20003660

up_dumpstate: stack base: 200037a0

up_dumpstate: stack size: 000007e4

up_stackdump: 20003660: 00000000 08000ff5 08000fb5 20000704 20003690 08000e1d 00000000 20003690

up_stackdump: 20003680: 200000b4 00000000 20000104 080004f3 200036d8 00000000 00000000 200000b4

up_stackdump: 200036a0: 00000000 20000104 20000138 200000c0 000000ef 2000064c 00000004 506fd680

up_stackdump: 200036c0: 506fd682 00000000 20000130 200000c0 08010dee 21000000 00002220 00000000

up_stackdump: 200036e0: 200000b4 20000648 40016818 00000001 002e0017 00000000 00000000 0801120d

up_stackdump: 20003700: 00000001 20003800 00000000 00000000 00000000 00000000 00000000 0800fa19

up_stackdump: 20003720: 08012d9c 00000000 00000000 08009c9f 6520514d 200037d0 00000000 08012d9d

up_stackdump: 20003740: 00000000 20001c70 00000000 00000000 00000000 00000000 00000000 00000000

up_stackdump: 20003760: 00000000 0800f1c9 00000000 74617473 08004601 00000101 20002c00 0800d17d

up_stackdump: 20003780: 00000101 0800460b 00000101 0800195f 00000000 00000000 00000000 00000000

up_registerdump: R0: 00000004 506fd680 506fd682 00000000 00000000 200000b4 00000000 20000104

up_registerdump: R8: 20000138 200000c0 000000ef 2000064c 20000130 200036d8 200000c0 08010dee

up_registerdump: xPSR: 21000000 PRIMASK: 00000000 CONTROL: 00000000

```

8 Mayıs 2018 Salı 17:47:56 UTC+3 tarihinde Nurettin SENYER yazdı:

8 Mayıs 2018 Salı 16:17:35 UTC+3 tarihinde Alan Carvalho de Assis yazdı:

Hi Nurettin,

Hi Alan,

 

I don't know details about LTDC hardware of STM32F429, but I think you
will need to modify the file
nuttx/configs/stm32f429i-disco/src/stm32_lcd.c to work with your LCD.

I tried to but i could not work.

 

This contribution was done by Marco Krahl, he is who knows more about it. 

[...] 

[Alan Carvalho de Assis]

Hi Nurettin,

What function is at 08010dee ?

BR,

Alan

On 5/8/18, Nurettin SENYER <nuretti...@gmail.com> wrote:

[Nurettin SENYER]

2018-05-08 18:26 GMT+03:00 Alan Carvalho de Assis <aca...@gmail.com>:

Hi Nurettin,

Hi Alan,

 

What function is at 08010dee ?

Which command do i use for this purpose?

 

BR,

Nurettin, 

> email to nuttx+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.
>

--

nurettins

[patacongo]

 

What function is at 08010dee ?

Which command do i use for this purpose?

 
See http://www.nuttx.org/doku.php?id=wiki:howtos:cortexm-hardfault

[Alan Carvalho de Assis]

Please look this documentation:

http://nuttx.org/doku.php?id=wiki:howtos:cortexm-hardfault

BR,

Alan

On 5/8/18, Nurettin SENYER <nuretti...@gmail.com> wrote:

[Nurettin SENYER]

8 Mayıs 2018 Salı 18:27:00 UTC+3 tarihinde Alan Carvalho de Assis yazdı:

Hi Nurettin,

What function is at 08010dee ?

I think the function name is stm32_ili93414ws_recvword.

[patacongo]

I suspect that the version of nuttx that you ran arm-none-eabi-objdump on is not the same version that you had in FLASH at time of the failure.

[Nurettin SENYER]

9 Mayıs 2018 Çarşamba 00:28:56 UTC+3 tarihinde patacongo yazdı:

I suspect that the version of nuttx that you ran arm-none-eabi-objdump on is not the same version that you had in FLASH at time of the failure.

How to check this?

Our nuttx version is nuttx-7.9.

[patacongo]

I suspect that the version of nuttx that you ran arm-none-eabi-objdump on is not the same version that you had in FLASH at time of the failure.

How to check this?

You burn the code that you compiled into FLASH

Our nuttx version is nuttx-7.9.

That is very old!  I release code every 3 months and that is 15 releases behind  from way back to April of 2015.  Not much that is said in this forum applies to anything that old.

[Gábor Kiss-Vámosi]

Hi,

The strange thing is you did minor changes and got hard fault.

It seems to me you only changed "configs/stm32f429i-disco/include/board.h"
Is it still working with the original version? If yes, then you can add the your modifications step-by-step to see which breaks the system.

[patacongo]

Our nuttx version is nuttx-7.9.

That is very old!  I release code every 3 months and that is 15 releases behind  from way back to April of 2015.  Not much that is said in this forum applies to anything that old.

And how could you run Lvgl?  You mention a "custom" version.  it must be very custom since Lvgl depends on the frame buffer driver that was first introduced in  September 2017????

[Marco]

Hi Greg,

i have implemented the ltdc and dma2d drivers for the stm32f429i in 2015/2016. At this time nuttx did not provide a clear guideline how to handle accessibility of additional functionality (graphic acceleration) by userspace (application level), provided by the underlying hardware.

Therefore an ltdc interface has been exported to provide the functionality at application level, e.g. for testing by the ltdc example within apps.

This example has been removed in 2017, because of direct acessing the kernel by the ltdc interface. This is absolutely correct within the nuttx os design rules. But this also has been broken down the access to the ltdc and dm2ad hardware.

Today, nuttx supports fb interface and handling by ioclt for basic interactions with the framebuffer driver and mmap to write to the memory region which is shared by the hardware controller and of course the graphic driver. I'm familar with this because this is also done by linux.

However, im willing to make the ltdc and dma2d controller accessible by posix interface using ioctl and mmap. But there are some additional information missing:


What provides stm32f429 (7xx)?

1. A ltdc controller with 2 independent layer. Both layer will be permanently blended and have the capability to be configured for different size, rgb types, alpha value and color key.

I think both layer could be accessible by /dev/fb0 and /dev/fb1. This is the way e.g. by nxp i.mx6 to operate with the foreground and background layer provided by the vpu/ipu. (linux os)

But for controlling such layer ioctl definitions are missing in nuttx fb interface specification. I think this additional ioctl part must be in platform specific header accessible by the application.
Which location/directory?


2. A dma2d controller to blend free layers. Free means any memory location except ccm. dma2d has the capability to be configured for alpha blending of different layer of different size and different rgb types (e.g. blend RGB565 with RGB24). Furthermore the result can be transferred to a third destination layer. This all is done via dma. (the name dma2d ;)).

What kind of device should be the dma2d controller?

Any fb device or /dev/dma2d?

How is it possible to create dynamic layers of different types (size, rgb) by the application to interact with the dma2d driver/controller?
Layer memory allocation must be done within the dma2d driver and can be transfered/copied by passing predefined structures with the ioclt argument by the application?

What do you think?

Marco

[patacongo]

i have implemented the ltdc and dma2d drivers for the stm32f429i in 2015/2016. At this time nuttx did not provide a clear guideline how to handle accessibility of additional functionality (graphic acceleration) by userspace (application level), provided by the underlying hardware.

Therefore an ltdc interface has been exported to provide the functionality at application level, e.g. for testing by the ltdc example within apps.

This example has been removed in 2017, because of direct acessing the kernel by the ltdc interface. This is absolutely correct within the nuttx os design rules. But this also has been broken down the access to the ltdc and dm2ad hardware.

Yes, I had to remove the code because it does not comply with POSIX OS interface standards.  And as with most compliant examples, it replicates like rabbits all over the system.  The STM32 F7 copied the LTDC and DMA2D so that the problem just continued to grow.  So it had to be excised to prevent further corruption of the interfaces.

There is no feature on the face of the planet that is sufficiently important to ever violate the strict POSIX interface.  That strict POSIX interface is the core of what NuttX is!  I need to be more diligent to prevent that from happening in the future.

The code is still in git history, of course, and is also available for easy review in in the Obsoleted repository.

Today, nuttx supports fb interface and handling by ioclt for basic interactions with the framebuffer driver and mmap to write to the memory region which is shared by the hardware controller and of course the graphic driver. I'm familar with this because this is also done by linux.

However, im willing to make the ltdc and dma2d controller accessible by posix interface using ioctl and mmap. But there are some additional information missing:

Many of the features are unique to the STM32 F4/F7 so even generic ioctl commands are not a proper solution.  I have thought about adding device-specific MCU commands.  Those would be defined in some file like arch/arm/include/stm32*/fb.h.  The existing framebuffer driver's ioctl command would simply forward all unrecognized IOCTL commands to the device-specific layer.

Using device-specific IOCTL commands, of course, is not a general solution.  But we are not talking about any kind of general solution anyway.  We are talking about a solution specific to the STM32F429 and simpler F7's.  No common graphics logic could ever use any logic that cannot be represented in a generic way.  No common graphics logic could ever use a device specific /dev/dma2d.  So from the point of view of the OS, it is all worthless throw away code if the interfaces cannot be represented in a device-independent way.  So my enthusiasm is low.

Greg

[patacongo]

It might be worth studying how Linux manages alpha blending with its framebuffer driver.  The NuttX and Linux framebuffer drivers are conceptually simply (although they actually share no specific details).

Maybe lilke: https://lists.yoctoproject.org/pipermail/meta-freescale/2014-November/011329.html

Or maybe: https://developer.toradex.com/knowledge-base/using-multiple-layers-and-alpha-blending-on-vybrid-linux

Or: https://www.linuxtv.org/downloads/v4l-dvb-apis-old/vidioc-g-fbuf.html

[patacongo]

Also, I would be happy to help extend the frame buffer driver definition if necessary.

Currently, the F429 interface is implemented as separate framebuffers, distinguished by a display number.

Perhaps there should be a single framebuffer with overlay planes?  (The framebuffer interface supports multple color planes, now but that is a different thing.  some older graphics hardware separated RGB or CYMK into separate frames.  I have not seen hardware like that in years).  That would allow the interface to manage interations between overlays.

This is the kind of thing that I can help you with if you can come up with a good interface description.

Greg

[Marco]

Hi Greg,

i'm familiar with the v4l2 api and have some small experience with video capturing from usb video class devices on linux.

From my perspective v4l2 api provides the best flexibility to handle such graphic controller (stm32f429/7xx).

Especially non destructive and destructive overlay corresponds to ltdc and dma2d functionality.

Is there any plan to add a v4l2 or v4n interface to nuttx?

Providing the v4l2 (v4n) api by the stm32f429 could be a good implementation example.

Marco

[patacongo]

Is there any plan to add a v4l2 or v4n interface to nuttx?

No, there is no plan for anything like that in place; I am the other person here and I am not planning to do it.  Plans could be made if people want to contribute and drive the effort.

Greg

[Marco]

Using planeinfo for that seems to be a good starting point.

Okay, I will take a look in the code an check what is also necessary than the planeinfo, before i suggest an interface extension.

Marco

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[patacongo]

Okay, I will take a look in the code an check what is also necessary than the planeinfo, before i suggest an interface extension.

I think an overly plane would be treated like a color plane, but they are conceptually different things:  https://en.wikipedia.org/wiki/Planar_(computer_graphics)

We could augment the interface so that it supports both color planes and overlays (like doubly indexed planes).  Or, as I mentioned, planar color is probably obsolete and perhaps we could just replace the plane terminology with overlay terminology?

Greg

[Marco]

Augmenting the interface is fine.

Suggestion:

Expanding fb_videoinfo_s to:

 struct fb_videoinfo_s

  {

    uint8_t    fmt;

    fb_coord_t xres;

    fb_coord_t yres;

    uint8_t    nplanes;

    uint8_t    noverlay; /* Number overlays */

 };

Then we need an additional structure e.g. struct fb_overlayinfo_s equal to fb_planeinfo_s plus maybe additional necessary information accessible by ioctl command FBIOGET_OVERLAYINFO.

This will hopefully not break down existing code and doesn't require to implement that interface when not supported by the hardware or driver.

But there are farther two existing interfaces /dev/fb0-0 and /dev/fb0-1 right?

[patacongo]

Suggestion:

Expanding fb_videoinfo_s to:

Seems reasonable.  Would you like me to create a branch in the upstream repository?

[patacongo]

I see I did not respond to your last question:

But there are farther two existing interfaces /dev/fb0-0 and /dev/fb0-1 right?

I don't have a definitive answer but I can offer some thoughts.

There is only /dev/fb0 at present.

There is logic in 'int fb_register(int display, int plane)'  to create multiple device.  That is mostly place holder logic and it assumes that each color plane would be represented as a separate device as /dev/fb0.n where .n is the color plane.  But that does not really make sense.  The plane parameter should be ideally be removed.  The plane number is used in IOCTL handling and I think it should be designed away or replaced with IOCTL arguments.

Certainly we could access all frame and overlay information using an open file descriptor for one framebuffer device.

The critical thing is how mmap() works.  mmap takes the file descriptor and returns the framebuffer address (mapped for access in user space if needed).  I don't think that the semantics of mmap() would support multiple overlays on the same device.  That would be the only argument for multiple devices for overlays and color planes.

NOTE: The framebuffer address is available in the planeinfo as well,  However, I think we have to stick with the semantics of mmap() to handle non-FLAT address environments where framebuffers need to be mapped into the user address space with an MPU or MMU.

Greg

[Marco]

With pleasure. Please send me account via mail when not shared for everyone.

Marco

[patacongo]

Hi, Marco,

Seems reasonable.  Would you like me to create a branch in the upstream repository?

With pleasure. Please send me account via mail when not shared for everyone.

I have created the branch: https://bitbucket.org/nuttx/nuttx/branch/overlay

You can fork that branch. If you want direct write access, I will need to add you to the bitbucket team using your Bitbucket username.  You can send me the bitbucket info by email.

Greg

[Marco]

The critical thing is how mmap() works.  mmap takes the file descriptor and returns the framebuffer address (mapped for access in user space if needed).  I don't think that the semantics of mmap() would support multiple overlays on the same device.  That would be the only argument for multiple devices for overlays and color planes.

NOTE: The framebuffer address is available in the planeinfo as well,  However, I think we have to stick with the semantics of mmap() to handle non-FLAT address environments where framebuffers need to be mapped into the user address space with an MPU or MMU.

Hi Greg,

using offsets when mapping the overlay memory to the application layer could be a straightforward solution.

The first overlay 0 has the offset 0. The offset of overlay 1 results by width * height * bpp of the overlay 0 and so on.

Sure, this requires that the entire overlay memory must be in a continguous address space. But that also complicates the access to the overlay memory using mmap.

Otherwise we need a file descriptor for each overlay. But control of all overlays can be done by each file descriptor.

Both solutions are not the best i think.

Marco

[patacongo]

using offsets when mapping the overlay memory to the application layer could be a straightforward solution.

The first overlay 0 has the offset 0. The offset of overlay 1 results by width * height * bpp of the overlay 0 and so on.

Actually, the size should byte the stride * height.  The stride is the number of bytes in a row which may not be the same as width * bpp / 8.

Sure, this requires that the entire overlay memory must be in a continguous address space. But that also complicates the access to the overlay memory using mmap.

It is a little kludgey, but the driver could interpret the offset I suppose, if it were available.

Currently, that offset is not provided with the FIOC_MMAP IOCTL.  That IOCTL is used in other drivers as well.  Also in the ROMFS file system.

Otherwise we need a file descriptor for each overlay. But control of all overlays can be done by each file descriptor.

Both solutions are not the best i think.

Agreed.

Greg

[patacongo]

using offsets when mapping the overlay memory to the application layer could be a straightforward solution.

The first overlay 0 has the offset 0. The offset of overlay 1 results by width * height * bpp of the overlay 0 and so on.

How about an IOCTL command to select the overlay.  After this IOCTL command, all overlay related operations (including mmap()) would use the selected overlay transparently with no other change to use usage semantics.

Greg

[Marco]

Yes, that sounds good. It resolves the disadvantages of the other both solutions. Agreed.

Another question.

During the interface design i come to the conclusion it would be fine to have two or three additional ioctl commands.

The first to create a virtual framebuffer device. This could be useful to access memory managed by the graphic controller/hardware, e.g. by stm's dma2d.

The second/third to control capabilities of the overlay or overlay operations.

But this also ensure the interface must be expanded for future graphic operations. The question here is? What kind of hardware behaves like a framebuffer?

Perhaps it is smart to define another interface like dri/drm or v4l for such devices/controller.

Marco

 

[patacongo]

During the interface design i come to the conclusion it would be fine to have two or three additional ioctl commands.

The first to create a virtual framebuffer device. This could be useful to access memory managed by the graphic controller/hardware, e.g. by stm's dma2d.

I don't fully understand what you mean be creating  a device.

The second/third to control capabilities of the overlay or overlay operations.

But this also ensure the interface must be expanded for future graphic operations. The question here is? What kind of hardware behaves like a framebuffer?

Look at drivers/lcd/lcd_framebuffer.c.  That adapts a serial or parallel LCD interface to provide a framebuffer.  It looks like any other framebuffer device, but must support a display update IOCTL command.

Perhaps it is smart to define another interface like dri/drm or v4l for such devices/controller.

I don't have enough insight to respond.

Greg

[Marco]

I mean, the first ioctl command to create a virtual overlay.

[patacongo]

During the interface design i come to the conclusion it would be fine to have two or three additional ioctl commands.

But this also ensure the interface must be expanded for future graphic operations. The question here is? What kind of hardware behaves like a framebuffer?

Look at drivers/lcd/lcd_framebuffer.c.  That adapts a serial or parallel LCD interface to provide a framebuffer.  It looks like any other framebuffer device, but must support a display update IOCTL command.

There are also several MCUs that support framebuffer hardware without overlays (LPC1788, LPC54, and others) as well as with overlays (DM320, SAMA5Dx).

There is also graphics hardware for which there is not yet any driver:  Raspi, i.MX6, i.MX RT.

[Marco]

Let me explain from the application perspective.

Usually nuttx will be configured with a fixed number of framebuffer and soon overlays and their size.

The fb devices are defined at compile time.

At application level it is useful to have the possibility to dynamically create allocate e.g memory regions (usually grapical objects) of different size and pass them to functions (graphic library) to perform common operations, e.g. blit or blend operation. This requires, among other things like setup regions of source and destination layer which should be blitted or e.g. set the layers alpha value.

I'll tell you nothing new.

This is especially useful to create several areas within a screen and blit the final image to the screen.

This all is supported by the stm32 hardware ltdc and dma2d controller. The hardware is specific, the functionality is common in graphical programming.

With two fixed size overlays the flexibility for such operations is very limited.

So i'm looking for a smart way how to provide such operations with a file descriptor of a framebuffer device, mmap and ioctl commands.

Providing simple overlay support of two layers (e.g. supported by stm32 ltdc) seems to be not complicated. I think I have to concentrate on this first.

Marco

[patacongo]

Allocation of frame buffers is difficult on some platforms, especially platforms with MMUs and vitualized heaps.

Normally, these systems have a large SDRAM and framebuffers are reserved at one end or the other of the SDRAM at link time.

So I would hope that the option to use pre-allocated overlay framebuffers is also supported.  They will be needed.

[Marco]

Sure, pre-allocated overlay framebuffers also supported. Dynamic framebuffer allocation is currently done in dma2d driver by nuttx kmalloc. dma2d accepts each memory except the core coupled (ccm).

[Marco]

Hi,

a short update by me.

I'm currently porting existing ltdc and dma2d driver stuff to the new  framebuffer interface with overlay support. I will replace the existing ltdc and dma2d code completely. Otherwise the code will be to complicated and not maintainable.

I see no disadvantage in this because the existing interface can't be used by application code anymore. And i think it isn't used by any other instance.

Ok, that will be a bit more work. I also have to  implement a small application for overlay testing.

Does anybody see any problems?

Marco

--
You received this message because you are subscribed to the Google Groups "NuttX" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nuttx+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[patacongo]

Does anybody see any problems?

No, that seems reasonable.  You are right, in its current state it certainly could not be destabilizing to anyone's effort.

There are no real "problems" that I see, but there is an issue:  The STM32F769 (arch/arm/src/stm32f7 and configs/stm32f769i-disco) took the LTDC and DMA2C from the STMF429i some months back.  The code differs very little from the STM32F429i.

The issue is that this, of course, should also be made consistent with the new interface.  Of course, it would be best to do this now since the code is (nearly) identical to the STM32F429i.

The SAMA5Dx (and the venerable DM32) also should have the change, but that is different hardware and so not so critical.

Perhaps as you make changes I could backport them to the STM32F769?  I don't have any hardware or anyway to verify the changes, however.

Greg

[Marco]

Hi Greg

I don't have any related hardware here. I took a look in the f769 reference manual in the past. Both cpus (f769 and f429) provides the same register set used by the LTDC and DMA2D controller. So the functionality seems identical. Should both share the corresponding code? I think more about to copy and modify. However, we can decide that when the f429 port is complete.

Marco

[patacongo]

I don't have any related hardware here. I took a look in the f769 reference manual in the past. Both cpus (f769 and f429) provides the same register set used by the LTDC and DMA2D controller. So the functionality seems identical. Should both share the corresponding code? I think more about to copy and modify. However, we can decide that when the f429 port is complete.

No, I never share peripherals across processor families.  That would be a build/maintenance nightmare!  They may be temporarily the same, but surely they will diverge in the future.  The STM32 F7 has a data cache which causes subtle differences in all peripherals that you DMA.