open-amp and DMA in interrupt mode

[wuampa...@gmail.com]

Hello all,

I've recently started to work with open-amp on a two-core Zynq 7020 with the intention of running Linux on CPU0 and bare-metal on CPU1. I was able to run the examples found in UG1186, but I'm finding many problems to run a more complex application, mainly due to my ignorance on this subject.

To start with, as suggested, I created an open-AMP boot placing the kernel base address on 0x10000000. Which is the reason for this? Are the addresses 0x0 to 0x10000000 never used?

Then, I placed the netboot offset address on 0x20000000, because leaving the address specified (0x11000000) did not allow Linux to boot. I don't know what this modification implies, but it works.

My intention is to run in CPU1 a bare-metal code that captures some data and stores it in an external 1-GB DDR via DMA in interrupt mode. This code, based on Xilinx answer AR# 57562, used to work when running on single CPU without open-amp. The first problem I found when trying to run it in the remote processor was the memory allocation since I was using the malloc instruction. Leaving them made Linux in CPU0 crash once the application in CPU1 was started. I replaced them with direct assignments to memory as for instance:

#define FRAME_BUFFER_BASEADDR 0x3e350000
u32 *fb0 = (u32 *)FRAME_BUFFER_BASEADDR;

It worked, although I'm not confident this is a good solution (although I place them at the end of the available memory for CPU1, I'm worried about code corruption).

In any case, I'm still not able to run the application on CPU1. Everytime I run it, Linux crashes (it doesn't answer anymore). It seems that CPU1 is working as expected (I have checked it using some LEDs of the board) but CPU0 stops. Apparently, the problem is on the call "XScuGic_CfgInitialize" that initializes the interrupt controller. If I comment it, Linux does not crash, but the application is not able to work properly. Any clue what may be happening?

Finally, I have edited system-conf.dtsi as shown below. Among other things, I'm afraid the S2MM interrupt may not be properly configured (I've not been able to find a suitable tutorial for this but some isolated comments and suggestions on the web). I pretend to use "dmabuf" to store the data from the DMA (using the bare-metal application). This buffer should also be seen by Linux, so that CPU0 and CPU1 talks each other for write and read operations. In Linux space I have the following code:

#define DDR_SIZE (0x7FFFFF)
#define DDR_BASE (0x3e800000)
.
.
.
mem_file = open("/dev/mem", O_RDWR);
ddr_buffer = mmap(NULL, DDR_SIZE, PROT_READ|PROT_WRITE, MAP_SHARED, mem_file, DDR_BASE );

I have not get to this point yet, but since I'm not confident in what I'm doing I prefer to show it to you:

/include/ "system-conf.dtsi"
/ {
reserved-memory {
#address-cells = <1>;
#size-cells = <1>;
ranges;
rproc_0_reserved: rproc@3e000000 {
no-map;
reg = <0x3e000000 0x01000000>;
};
};

amba {
/* "mmio-sram" is the memory for the firmware*/
elf_ddr_0: ddr@0 {
compatible = "mmio-sram";
reg = <0x3e000000 0x400000>;
};
};

remoteproc0: remoteproc@0 {
compatible = "xlnx,zynq_remoteproc";
interrupt-parent = <&intc>;
/* S2MM is interrupt 61-32=29 */
interrupts = <0 29 4>;
firmware = "ADC2DMACPU1.elf";
vring0 = <15>;
vring1 = <14>;
/* firmware memory specidied here */
srams = <&elf_ddr_0>;
};

dmabuf {
compatible = "dmem-uio";
reg = < 0x3e800000 0x7FFFFF>;
};

};

[Jiaying Liang]

> -----Original Message-----
> From: open...@googlegroups.com [mailto:open...@googlegroups.com]
> On Behalf Of wuampa...@gmail.com
> Sent: Tuesday, March 06, 2018 3:54 AM
> To: open-amp <open...@googlegroups.com>
> Subject: [open-amp] open-amp and DMA in interrupt mode
>
> Hello all,
>
> I've recently started to work with open-amp on a two-core Zynq 7020 with
> the intention of running Linux on CPU0 and bare-metal on CPU1. I was able
> to run the examples found in UG1186, but I'm finding many problems to run
> a more complex application, mainly due to my ignorance on this subject.
>
> To start with, as suggested, I created an open-AMP boot placing the kernel
> base address on 0x10000000. Which is the reason for this? Are the addresses
> 0x0 to 0x10000000 never used?

[Wendy] This is not the requirement anymore, it was because before the CPU1 firmware
Runs from 0, in this case, you will need to place the kernel to a higher address. If your CPU1
Runs from higher address, you don't need to do it.

>
> Then, I placed the netboot offset address on 0x20000000, because leaving
> the address specified (0x11000000) did not allow Linux to boot. I don't know
> what this modification implies, but it works.

[Wendy] You don't need to have this configuration, the normal Zynq boot flow should work.

>
> My intention is to run in CPU1 a bare-metal code that captures some data
> and stores it in an external 1-GB DDR via DMA in interrupt mode. This code,
> based on Xilinx answer AR# 57562, used to work when running on single CPU
> without open-amp. The first problem I found when trying to run it in the
> remote processor was the memory allocation since I was using the malloc
> instruction. Leaving them made Linux in CPU0 crash once the application in
> CPU1 was started. I replaced them with direct assignments to memory as for
> instance:
>
> #define FRAME_BUFFER_BASEADDR 0x3e350000
> u32 *fb0 = (u32 *)FRAME_BUFFER_BASEADDR;

[Wendy] You will need to make sure it is not overlapped with the Linux kernel
System memory as you want to predefined the DMA buffers.

>
> It worked, although I'm not confident this is a good solution (although I
> place them at the end of the available memory for CPU1, I'm worried about
> code corruption).
>
> In any case, I'm still not able to run the application on CPU1. Everytime I run
> it, Linux crashes (it doesn't answer anymore). It seems that CPU1 is working
> as expected (I have checked it using some LEDs of the board) but CPU0 stops.
> Apparently, the problem is on the call "XScuGic_CfgInitialize" that initializes
> the interrupt controller. If I comment it, Linux does not crash, but the
> application is not able to work properly. Any clue what may be happening?

[Wendy] did you use "-DUSE_AMP=1" when you compile the CPU1 application?
If you don't define this flag, when you use XScuGic_CfgInitialize(), it will reinitialize
the GIC which will reset the GIC settings done by Linux.

>
> Finally, I have edited system-conf.dtsi as shown below. Among other things,
> I'm afraid the S2MM interrupt may not be properly configured (I've not been
> able to find a suitable tutorial for this but some isolated comments and
> suggestions on the web). I pretend to use "dmabuf" to store the data from
> the DMA (using the bare-metal application). This buffer should also be seen
> by Linux, so that CPU0 and CPU1 talks each other for write and read
> operations. In Linux space I have the following code:
>
> #define DDR_SIZE (0x7FFFFF)
> #define DDR_BASE (0x3e800000)

[Wendy] It looks like you want to use "/dev/mem" to access your DMA buffers.
Are you using Xilinx AXI DMA? Will you access the DMA controller from userspace?
Or you just want to access the buffers from userspace?

> --
> You received this message because you are subscribed to the Google Groups
> "open-amp" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to open-amp+u...@googlegroups.com.
> To post to this group, send an email to open...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[wuampa...@gmail.com]

Thank you very much for your answers. Regarding your questions:

- did you use "-DUSE_AMP=1" when you compile the CPU1 application?

Yes, I did. I added "-DUSE_AMP=1 in the extra_compiler_flags, along with other modifications described in UG1186.

- Are you using Xilinx AXI DMA?

Yes.

- Will you access the DMA controller from userspace or you just want to access the buffers from userspace?

DMA is controlled by the bare-metal application. From Linux userspace, I only want to access the buffers, not the DMA controller

[Jiaying Liang]

> -----Original Message-----
> From: open...@googlegroups.com [mailto:open...@googlegroups.com]
> On Behalf Of wuampa...@gmail.com
> Sent: Tuesday, March 06, 2018 9:39 AM
> To: open-amp <open...@googlegroups.com>

> Subject: [open-amp] Re: open-amp and DMA in interrupt mode
>
> Thank you very much for your answers. Regarding your questions:
>
> - did you use "-DUSE_AMP=1" when you compile the CPU1 application?
>
> Yes, I did. I added "-DUSE_AMP=1 in the extra_compiler_flags, along with
> other modifications described in UG1186.

[Wendy] have you debugged it? it sounds like when the interrupt is getting handled on core1, it causes the Linux corruption.
You can try un
Connect to core1 with xsdb through jtag, set the breakpoint, and then launch the application and then see at which point the core1 corrupts core0.

>
> - Are you using Xilinx AXI DMA?
>
> Yes.
>
> - Will you access the DMA controller from userspace or you just want to
> access the buffers from userspace?
>
> DMA is controlled by the bare-metal application. From Linux userspace, I
> only want to access the buffers, not the DMA controller

[Wendy] ok, in this case, you can access use /dev/mem or UIO to access the DMA buffers.

[wuampa...@gmail.com]

I finally managed to debug the code in CPU1 when using openAMP (which is quite tricky, by the way). I've seen that everything's fine until the code gets to the "while" were the app waits for events from CPU0. It is the same code as in the echo test example:

...
1 LPRINTF("Waiting for events...\n");
2 while(1) {
3 hil_poll(proc->proc, 0);

/* we got a shutdown request, exit */
4 if (evt_chnl_deleted) {
5 break;
}

6 if (evt_virtio_rst) {
/* vring rst callback, reset rpmsg */
7 LPRINTF("De-initializing RPMsg\n");
8 rpmsg_deinit(proc->rdev);
9 proc->rdev = NULL;

10 LPRINTF("Reinitializing RPMsg\n");
11 status = rpmsg_init(hproc, &proc->rdev,
12 rpmsg_channel_created,
13 rpmsg_channel_deleted,
14 rpmsg_read_cb,
15 1);
16 if (status != RPROC_SUCCESS) {
17 LPERROR("Reinit RPMsg failed\n");
18 break;
}
19 LPRINTF("Reinit RPMsg succeeded\n");
20 evt_chnl_deleted=0;
21 evt_virtio_rst = 0;
}
}

When the debugger is on line 3 and I proceed with next step, in the Linux terminal appears the sentence:

"virtio_rpmsg_bus virtio0: creating channel rpmsg-openamp-demo-channel addr 0x1"

When the debugger is on line 4 and I proceed with next step, the Linux terminal stops answering. CPU1 cannot be commanded from SDK as well. SDK shows CPU1 as running, but if I try to pause the app it says that CPU1 cannot be halted. Only after restarting the system I can recover the control of CPU1 (and CPU0, of course).

If I comment the call "XScuGic_CfgInitialize" then the behaviour is the expected one, that is, the debug goes from line 3 -> 4 -> 6 -> 3 -> 4 -> 6 and then goes to _poll (which is the same behaviour as in the echo test example).

[Jiaying Liang]

> -----Original Message-----
> From: open...@googlegroups.com [mailto:open...@googlegroups.com]
> On Behalf Of wuampa...@gmail.com
> Sent: Wednesday, March 07, 2018 7:23 AM
> To: open-amp <open...@googlegroups.com>
> Subject: [open-amp] Re: open-amp and DMA in interrupt mode
>

[Wendy] did you use -O0 for compiling your application?
You can step into hill_poll() to see what it does after it receives the message.
You can break your rpmsg_cb() in your application and check if you are
Reading /writing with the expected shared memory.

[wuampa...@gmail.com]

Thank you very much for your suggestion. By the way, based on other related threads, I have also detected some issues I was not considering:

1. First, I have added "XScuGic_InterruptMaptoCpu(p_intc_inst, CPU_ID, s2mm_intr_id);" after "XScuGic_CfgInitialize" to map the interrupt to CPU1.

2. I've replaced mallocs by "metal_allocate_memory" instead of direct allocation (which I didn't like at all).

3. The last thing I want to do is something I've seen in this thread (probably written by you): https://groups.google.com/forum/#!msg/open-amp/6qxPSKaS02I/jMdOPa3IAAAJ

Considering I'm registering a UIO dmem device:

dmabuf {
compatible = "dmem-uio";
reg = < 0x3e800000 0x7FFFFF>;
};

It is said that, "On the remote side, you will need to map those memory as non-cacheable memory from initialization."

To do this, it seems that boot.S should be edited by something similar to:

/* In case of AMP, map virtual address 0x3e000000 to 0x3f000000 and mark it as non-cacheable */
#if USE_AMP==1
ldr r3, =x1ff /* 512 entries to cover 512MB DDR */
ldr r0, =TblBase /* MMU Table address in memory */
add r0, r0, #0x800 /* Address of entry in MMU table, for 0x20000000 */
ldr r2, =0x0c02 /* S=b0 TEX=b000 AP=b11, Domain=b0, C=b0, B=b0 */
mmu_loop:
str r2, [r0] /* write the entry to MMU table */
add r0, r0, #0x4 /* next entry in the table */
add r2, r2, #0x100000 /* next section */
subs r3, r3, #1
bge mmu_loop /* loop till 16MB is covered */
#endif

But I don't know how to edit it for the addresses 0x3e800000 0x7FFFFF...

Finally, in the bare-metal code, how can I say to the DMA to write in this part of the memory?
What I'm doing it now is to define a pointer to this part part of the memory and provide it to the DMA:

cv_buf = (int *)0x3e800000;
memset(rcv_buf, 0, SAMPLES_PER_FRAME*NUMBER_OF_FRAMES*sizeof(int));

This was suitable when running the code in CPU0, but probably should be done in a different way now...

[Jiaying Liang]

> -----Original Message-----
> From: open...@googlegroups.com [mailto:open...@googlegroups.com]
> On Behalf Of wuampa...@gmail.com
> Sent: Thursday, March 08, 2018 8:51 AM
> To: open-amp <open...@googlegroups.com>
> Subject: [open-amp] Re: open-amp and DMA in interrupt mode
>

> Thank you very much for your suggestion. By the way, based on other
> related threads, I have also detected some issues I was not considering:
>
> 1. First, I have added "XScuGic_InterruptMaptoCpu(p_intc_inst, CPU_ID,
> s2mm_intr_id);" after "XScuGic_CfgInitialize" to map the interrupt to CPU1.

[Wendy] You will need to connect the interrupt to your baremetal application.
If you can make sure the ISR is called for the expected interrupt, you should have
Done your interrupt initialization correct.

>
> 2. I've replaced mallocs by "metal_allocate_memory" instead of direct
> allocation (which I didn't like at all).

[Wendy] I am not sure why you need to make this change in your application.

[Wendy] You will need to map it uncacheable, you can refer this the io_mem_map()
Implementation in https://github.com/OpenAMP/libmetal/blob/master/lib/system/generic/zynq7/sys.c
on how to do it.

Best Regards,
Wendy

> What I'm doing it now is to define a pointer to this part part of the memory
> and provide it to the DMA:
>
> cv_buf = (int *)0x3e800000;
> memset(rcv_buf, 0, SAMPLES_PER_FRAME*NUMBER_OF_FRAMES*sizeof(int));
>
> This was suitable when running the code in CPU0, but probably should be
> done in a different way now...
>