Place remote memory into OCM?

[Sven Schwermer]

Hi there,

when having a Linux master and a Bare Metal/RTOS remote on a Zynq, is
it possible to specify where in the system memory map the remote
memory (.data, .text, ...) will be placed? In my application the
remote needs to run in/from the OCM (mapped high) as it is much faster
than the DDR memory. I remember that in old versions of remoteproc the
Linux host would ignore the addresses from the linker file. Did that
change? And how do I specify this with OpenAMP?

Can somebody help me? Thanks!
Sven

[Jiaying Liang]

Hi Sven,

We have made some changes to the remoteproc for similar case for Zynq MPSoC and it is not upstreamed yet.
Here is the patch https://github.com/Xilinx/linux-xlnx/commit/944d47dbb5aa0433dde394c54406ae9c2cb4219a

You may want it for Zynq.

Just one thing, the shared memory needs to be in DDR since OCM memory is too small for the current implementation of rpmsg.

About where to specify the memory mapping, you can specify them in the resource table. Here is an example: https://github.com/OpenAMP/open-amp/blob/master/apps/samples/remote/baremetal/matrix_multiply/rsc_table.c

Best Regards,
Wendy

--
You received this message because you are subscribed to the Google Groups "open-amp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-amp+u...@googlegroups.com.
To post to this group, send an email to open...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately.

[schwer...@gmail.com]

Hi Wendy,

thanks for your response. I now got my remote processor core to boot up from OCM (0xfffc0000 through 0xffffffff). The rpmsg communication, however, is not working because the vring buffer is empty as the master tries to read it. When reading the buffer via JTAG with XMD/GDB the buffer is empty, too. So for some reason the remote processor core doesn't write to the (correct) vring buffer.

I am suspecting a cache policy issue to be the cause of this problem. However, the cache policies for the relevant address ranges should be the same:
* In the working non-OCM implementation the vring descriptors start from 0x08000000, the buffers from 0x08040000.
* In the non-working OCM implementation the vring descriptors start from 0x00000000, the buffers from 0x00040000.
* After inspecting the cache policy in the MMU table of the remote core, I found that the policy is 0x15de6 for the entire region 0x00000000 through 0x20000000.

Here is the relevant debug output:
$ modprobe virtio dyndbg==p
$ modprobe virtio_ring dyndbg==p
$ modprobe virtio_rpmsg_bus dyndbg==p
$ modprobe remoteproc dyndbg==p
$ modprobe zynq_remoteproc firmware=OpenAMP_OCM_echo-test.elf dyndbg==p
CPU1: shutdown
zynq_remoteproc 0.remoteproc: Using firmware: OpenAMP_OCM_echo-test.elf
remoteproc0: 0.remoteproc is available
remoteproc0: Note: remoteproc is still under development and considered experimental.
remoteproc0: THE BINARY FORMAT IS NOT YET FINALIZED, and backward compatibility isn't yet guaranteed.
remoteproc0: rsc: type 0
remoteproc0: carveout rsc: da fffc0000, pa fffc0000, len 40000, flags 0
remoteproc0: carveout va 90980000, dma 0, len 0x40000
remoteproc0: rsc: type 3
remoteproc0: rsc: type 0
remoteproc0: rsc: type 3
remoteproc0: rsc: type 0
remoteproc0: rsc: type 3
remoteproc0: vdev rsc: id 7, dfeatures 1, cfg len 0, 2 vrings
remoteproc0: vdev rsc: vring0: da 0, qsz 256, align 4096
remoteproc0: vdev rsc: vring1: da 4000, qsz 256, align 4096
virtio virtio0: reset !
virtio virtio0: status: 1
virtio_rpmsg_bus virtio0: status: 3
remoteproc0: vring0: va 91000000 dma 0 size 3000 idr 0
remoteproc0: vring0: va 91000000 qsz 256 notifyid 0
remoteproc0: vring1: va 91004000 dma 4000 size 3000 idr 1
remoteproc0: vring1: va 91004000 qsz 256 notifyid 1
remoteproc0: powering up 0.remoteproc
remoteproc0: Booting fw image OpenAMP_OCM_echo-test.elf, size 428243
zynq_remoteproc 0.remoteproc: iommu not found
remoteproc0: rsc: type 0
remoteproc0: rsc: type 3
remoteproc0: phdr: type 1 da 0xfffc0000 memsz 0x36f00 filesz 0x20014
zynq_remoteproc 0.remoteproc: zynq_rproc_start
remoteproc0: remote processor 0.remoteproc is now up
virtio_rpmsg_bus virtio0: buffers: va 91040000, dma 0x40000
Added buffer head 0 to 8dd77800
Added buffer head 1 to 8dd77800
# SNIP: I have taken the other buffers out of this log
Added buffer head 255 to 8dd77800
virtio_rpmsg_bus virtio0: status: 7
remoteproc0: kicking vq index: 0
zynq_remoteproc 0.remoteproc: KICK Firmware to start send messages vqid 0
virtio_rpmsg_bus virtio0: rpmsg host is online
zynq_remoteproc 0.remoteproc: KICK Linux because of pending message
remoteproc0: registered virtio0 (type 7)
remoteproc0: vq index 0 is interrupted
virtqueue callback for 8dd77800 (7f01c004)
virtio_rpmsg_bus virtio0: From: 0x0, To: 0x0, Len: 0, Flags: 0, Reserved: 0
rpmsg_virtio RX: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
virtio_rpmsg_bus virtio0: msg received with no recipient
Added buffer head 0 to 8dd77800
No more buffers in queue
virtio_rpmsg_bus virtio0: Received 1 messages
remoteproc0: kicking vq index: 0
zynq_remoteproc 0.remoteproc: KICK Firmware to start send messages vqid 0

[Jiaying Liang]

Hi Sven,

Please see my comments inline.

Thanks,

Wendy

-----Original Message-----
From: open...@googlegroups.com [mailto:open...@googlegroups.com] On Behalf Of schwer...@gmail.com
Sent: Tuesday, February 02, 2016 2:26 AM
To: open-amp
Subject: Re: [open-amp] Place remote memory into OCM?

Hi Wendy,

thanks for your response. I now got my remote processor core to boot up from OCM (0xfffc0000 through 0xffffffff). The rpmsg communication, however, is not working because the vring buffer is empty as the master tries to read it. When reading the buffer via JTAG with XMD/GDB the buffer is empty, too. So for some reason the remote processor core doesn't write to the (correct) vring buffer.

I am suspecting a cache policy issue to be the cause of this problem. However, the cache policies for the relevant address ranges should be the same:
* In the working non-OCM implementation the vring descriptors start from 0x08000000, the buffers from 0x08040000.
* In the non-working OCM implementation the vring descriptors start from 0x00000000, the buffers from 0x00040000.
* After inspecting the cache policy in the MMU table of the remote core, I found that the policy is 0x15de6 for the entire region 0x00000000 through 0x20000000

[Wendy] When putting the code in OCM, it runs faster. Could you just do a test, after the remote receives the interrupt, and before it kick the Linux, could you do flush cache to see if the issue still there? Or even just disable flush from remote.

[Sven Schwermer]

Hi Wendy, I got it working now -- without the cache needing to be flushed. I simply forgot to adjust the functions platform_vatopa() and env_map_vatopa(), respectively. I am currently working on a solution to have carveouts in both OCM and DDR memories. This will require some kernel changes though...

[Jiaying Liang]

Hi Sven,

 

Good to know you have got it working.

 

Could you share your modification to the mailing list?

 

Thanks,

Wendy

 

From: open...@googlegroups.com [mailto:open...@googlegroups.com] On Behalf Of Sven Schwermer
Sent: Wednesday, February 03, 2016 8:10 AM
To: open-amp
Subject: Re: [open-amp] Place remote memory into OCM?

 

Hi Wendy, I got it working now -- without the cache needing to be flushed. I simply forgot to adjust the functions platform_vatopa() and env_map_vatopa(), respectively. I am currently working on a solution to have carveouts in both OCM and DDR memories. This will require some kernel changes though...

--

You received this message because you are subscribed to the Google Groups "open-amp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-amp+u...@googlegroups.com.

To post to this group, send email to open...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Sven Schwermer]

Hi Wendy,

this is my hack to make it work. I am, however, not sure whether there are cases the DMA address and the physical address shouldn't match. I also don't know whether this hack works for the Zynq MPSoC chips.

Sven

[Jiaying Liang]

Hi Sven,

Thanks for sharing your kernel patch.

You also mentioned you need to change platform_vatopa() and env_map_vatopa(), do you mind sharing that change too?

Thanks,
Wendy

From: open...@googlegroups.com [mailto:open...@googlegroups.com] On Behalf Of Sven Schwermer
Sent: Thursday, February 04, 2016 12:27 AM
To: open-amp
Subject: Re: [open-amp] Place remote memory into OCM?

[Sven Schwermer]

Hi Wendy, please excuse the late reply!

I ended up changing it back. I now have .text and .data (and .bss) sections in both of the following two memory areas:

• 0x00000000 - 0x08000000
• 0xfffc0000 - 0xffffffff

So the Vrings start at 0x08000000 as intended by the example application. As a result the original address translation needs to be applied.

Sven

[Jiaying Liang]

Hi Sven,

 

Thanks, I am thinking if we can change the code to be more generic for the address mapping.

 

Best Regards,

[a.curtis]

Just one thing, the shared memory needs to be in DDR since OCM memory is too small for the current implementation of rpmsg.

About where to specify the memory mapping, you can specify them in the resource table. Here is an example: https://github.com/OpenAMP/open-amp/blob/master/apps/samples/remote/baremetal/matrix_multiply/rsc_table.c

Best Regards,
Wendy

Hello Wendy,

You mention that the current implementation of rpmsg does not allow the ring buffers to reside in OCM memory. However when I look at the size of the buffers as specified in the header files they look very small. Could you provide more information? 

Thanks

Allen 

[a.curtis]

Hello,

A lot of the information contained in this posting is out of date. Is there new information to run FreeRTOS AMP from OCM? Is there a new rsc_table.c to reference for memory mapping?

Thanks

Allen

[Jiaying Liang]

Hi Allen,

The rpmsg buffer size is 512, you can find it in OpenAMP repo: lib/include/openamp/rpmsg_core.h

And also hardcoded in the linux kernel: drivers/rpmsg/virtio_rpmsg_bus.c

 

The number of buffers are also hardcoded in the linux kernel: rpmsg/virtio_rpmsg_bus.c as MAX_RPMSG_NUM_BUFS which is 512.

 

That is the shared memory size is 512*512 per vring, there are 2 vrings, one for tx and the other for rx, it is totally 512 KB.

 

Best Regards,

Wendy

 

--

You received this message because you are subscribed to the Google Groups "open-amp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-amp+u...@googlegroups.com.

To post to this group, send email to open...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jiaying Liang]

Hi Allen,

 

Which SoC are you using?

 

Best Regards,

Wendy

 

From: open...@googlegroups.com [mailto:open...@googlegroups.com] On Behalf Of a.curtis

Sent: Tuesday, April 19, 2016 1:23 PM
To: open-amp

--

You received this message because you are subscribed to the Google Groups "open-amp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-amp+u...@googlegroups.com.

To post to this group, send email to open...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[a.curtis]

 

Which SoC are you using?

 

We are using the Zync7000 with a bunch of PL.

 

[a.curtis]

See inline below...

 

We have made some changes to the remoteproc for similar case for Zynq MPSoC and it is not upstreamed yet.
Here is the patch https://github.com/Xilinx/linux-xlnx/commit/944d47dbb5aa0433dde394c54406ae9c2cb4219a

This patch link is no longer valid. Could somone please provide either a new link or a copy of the patch? I need to verify whether my OpenAMP source contains these modifications.

 

About where to specify the memory mapping, you can specify them in the resource table. Here is an example: https://github.com/OpenAMP/open-amp/blob/master/apps/samples/remote/baremetal/matrix_multiply/rsc_table.c

Is there an example rsc_table.c for placing the remote (slave) in OCM?

TIA,

Allen

[Jiaying Liang]

HI Allen,

 

From: open...@googlegroups.com [mailto:open...@googlegroups.com] On Behalf Of a.curtis

Sent: Tuesday, April 19, 2016 4:15 PM
To: open-amp

Subject: Re: [open-amp] Place remote memory into OCM?

See inline below...

 

We have made some changes to the remoteproc for similar case for Zynq MPSoC and it is not upstreamed yet.
Here is the patch https://github.com/Xilinx/linux-xlnx/commit/944d47dbb5aa0433dde394c54406ae9c2cb4219a

 

This patch link is no longer valid. Could somone please provide either a new link or a copy of the patch? I need to verify whether my OpenAMP source contains these modifications.

[Wendy] The linke is valid to me, are you able to access https://github.com/Xilinx/linux-xlnx?

 

About where to specify the memory mapping, you can specify them in the resource table. Here is an example: https://github.com/OpenAMP/open-amp/blob/master/apps/samples/remote/baremetal/matrix_multiply/rsc_table.c

 

Is there an example rsc_table.c for placing the remote (slave) in OCM?

[Wendy] there is no example to put slave in OCM for Zynq. Assuming you have make your linker script able to build your application on OCM, you will need to change the carveout section in the resource table for your  firmware.

 

Details of resource table is here:

https://github.com/OpenAMP/open-amp/blob/master/lib/include/openamp/remoteproc.h

 

If you can fit the whole FreeRTOS in to OCM, you can update the CARVEOUT_SRC in your rsc_table.c. To

 

{ RSC_CARVEOUT, <FIRMWARE_START_ADDR>, <FIRMWARE_START_ADDR>, <FIRMWARE_LEN>, 0, 0, "ELF_COUT", },

 

 

Best Regards,

Wendy

 

TIA,

Allen

--
You received this message because you are subscribed to the Google Groups "open-amp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-amp+u...@googlegroups.com.

To post to this group, send email to open...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jiaying Liang]

Hi Allen,

 

The kernel remoteproc core patch is attached.

 

Best Regards,

Wendy

[a.curtis]

Wendy,

The link is working for me too now...? Thanks

I will try making the other modifications tomorrow....

[a.curtis]

Wendy,

The below link doesn't work. Perhaps I was confused about which link.

[Jiaying Liang]

Hi Allen,

 

https://github.com/OpenAMP/open-amp/blob/master/apps/machine/zynq7/rsc_table.c and https://github.com/OpenAMP/open-amp/blob/master/apps/machine/zynq7/rsc_table.h

 

If you are using Xilinx SDK, it is better to use 2016.1.

 

Best Regards,

Wendu

 

From: open...@googlegroups.com [mailto:open...@googlegroups.com] On Behalf Of a.curtis
Sent: Tuesday, April 19, 2016 5:40 PM
To: open-amp
Subject: Re: [open-amp] Place remote memory into OCM?

 

Wendy,

 

The below link doesn't work. Perhaps I was confused about which link.

 

About where to specify the memory mapping, you can specify them in the resource table. Here is an example: https://github.com/OpenAMP/open-amp/blob/master/apps/samples/remote/baremetal/matrix_multiply/rsc_table.c

 

--

You received this message because you are subscribed to the Google Groups "open-amp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-amp+u...@googlegroups.com.
To post to this group, send email to open...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Allen Curtis]

Wendy,

 

If you are using Xilinx SDK, it is better to use 2016.1.

I am new to Xilinx development. If I use Xilinx SDK v2016.1 does that mean I need to upgrade PetaLinux and Vivado to the same version? My guess is yes.

Thanks,
Allen

[Allen Curtis]

Wendy,

 

If you are using Xilinx SDK, it is better to use 2016.1.

 

I don't see a PetaLinux 2016.1. The toolchain must be:

• Xilinx SDK 2016.1
• Vivado 2016.1
• PetaLinux 2015.4

[Jiaying Liang]

HI Allen,

 

I think you can use 2016.1 SDK to create FreeRTOS OpenAMP app.

 

On the Linux side you can keep using 2015.4 PetaLinux for now.

Please note that HDF from Vivado 2016.1 will not work with PetaLinux 2015.4.

 

But for you test only, you may be able to just use the PetaLinux 2015.4 default reference BSP without using your own hardware project.

 

Best Regards,

Wendy

 

From: open...@googlegroups.com [mailto:open...@googlegroups.com] On Behalf Of Allen Curtis
Sent: Tuesday, April 19, 2016 9:17 PM
To: open...@googlegroups.com
Subject: Re: [open-amp] Place remote memory into OCM?

 

Wendy,
 

--

You received this message because you are subscribed to the Google Groups "open-amp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to

open-amp+u...@googlegroups.com.

To post to this group, send email to

open...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[a.curtis]

Hello Wendy,

I reviewed the patch. It looks like the source code in 2016.1 has all these changes with the exception of one minor difference. The memset_io() call at the end of this insertion block.

diff --git a/drivers/remoteproc/remoteproc_core.c b/drivers/remoteproc/remoteproc_core.c

index 497582c..3c04596 100644

--- a/drivers/remoteproc/remoteproc_core.c

+++ b/drivers/remoteproc/remoteproc_core.c

@@ -581,12 +585,26 @@ static int rproc_handle_carveout(struct rproc *rproc,

  return -ENOMEM;

  }

 

+#if RPROC_CARVEOUT_USE_IOREMAP

+ /*

+ * WORKAROUND for handle multiple memory regions by using ioremap

+ * This is a temporary solution until we have iommu/smmu ready

+ */

+ va = devm_ioremap_nocache(dev, rsc->pa, rsc->len);

+ if (IS_ERR(va)) {

+ dev_err(dev->parent, "Unable to map memory %08x size %08x\n",

+ rsc->pa, rsc->len);

+ ret = PTR_ERR(va);

+ goto free_carv;

+ }

+ /* In case of TCM, have to initialize the memory after ioremap,

+ * otherwise, when loading segments to the memory, kernel will

+ * report "Bad mode in Synchronous Abort handler detected" error.

+ */

+ memset_io(va, 0, rsc->len);

#else

Thanks

Allen

[Jiaying Liang]

Hi Allen,

 

Please note that that patch is for aarch64 only, you will need to make it also available for Zynq.

 

Best Regards,

Wendy

 

From: open...@googlegroups.com [mailto:open...@googlegroups.com] On Behalf Of a.curtis
Sent: Wednesday, April 20, 2016 2:00 PM
To: open-amp
Subject: Re: [open-amp] Place remote memory into OCM?

 

Hello Wendy,

--

You received this message because you are subscribed to the Google Groups "open-amp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-amp+u...@googlegroups.com.
To post to this group, send email to open...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.