ReconOS and Vivado

[Sebastian Meisner]

Hi Folks!

I'm on the way to port ReconOS to the Vivado tools, so that you can use plain
Vivado to generate a bitstream for your ZedBoard. However, i'm stuck at the
moment and i would like to have some ideas where to investigate further.

Here's the situation:

Currently i have ported all ReconOS components (memory controller, OSIF,
arbiter etc.) to Vivado IP Integrator. Further, i have a Vivado block design
with a complete and (hopefully) correctly wired ReconOS system. My design
synthesises happily into a bitstream and loads into the ZedBoard. However, the
sortdemo gets stuck.

This is what i figured out: OSIF works and the sortdemo hardware thread gets
the address of the data it shall sort: 0x000a45e8. When it wants to read the
data in STATE_READ it generates a MEMIF requests which passes through the
arbiter and reaches the MMU. The MMU then tries a page walk, since it does not
know yet how to translate the address. So the MMU generates another MEMIF
request and sends it to the reconos_memif_memory_controller.

And here begins the trouble: the memory controller sees the 4 byte request for
address 0x1dc00040. I guess that's the address of the page table? But there is
never an answer to this request and thus the whole system gets stuck while
doing the memory request.


Please see the attached waveform diagram. Do you have any suggestion on how to
solve or investigate into this problem?

Kind regards
Sebastian

[Guanwen (Henry) Zhong]

Hello Sebastian,

> I'm on the way to port ReconOS to the Vivado tools.

Nice. I am currently extending multiple memory controllers and arbiter for MMU that connects to proc_control on ReconOS. And I also plan to move reconos from EDK to Vivado this week. :-)

> I guess that's the address of the page table?

Yes, you're right, the address is the L2 page table address. The L1 page table address is come from MMU_Pgd in reconos_proc_control which is set by "reconos_proc_control_set_pgd(_proc_control)" of "reconos_init()" function in the user space program. With the L2 page table address, L1 page table address and virtual address, MMU can get the physical address then send it to memory controller again for reading data. The 2-level page table information is come from "page.h" in the linux kernel, which is the default setting.

> But there is never an answer to this request and thus the whole system gets stuck while 

doing the memory request. 

Are you doing a runtime simulation? (I mean doing the simulation on the real board running Linux). As you switch to Vivado, the memory addresses of each hw IP might be changed. If the addresses are changed, we need to modify the according base addresses in the driver also.

> Do you have any suggestion on how to solve or investigate into this problem? 

If you are running on Linux, I think you can print the L1 and L2 page table addresses (using "virt_to_phys" for L1 address, can check "case RECONOS_PROC_CONTROL_SET_PGD_ADDR" in "proc_control_ioctl()" function from linux/driver/proc_control.c) and see whether it matches your simulation.

Thanks,

Guanwen

[Sebastian Meisner]

Hi Guanwen,

thanks for your answer!

>> But there is never an answer to this request and thus the whole system gets
> stuck while
> doing the memory request.
> Are you doing a runtime simulation? (I mean doing the simulation on the real
> board running Linux). As you switch to Vivado, the memory addresses of each hw
> IP might be changed. If the addresses are changed, we need to modify the
> according base addresses in the driver also.

Yes, i do a runtime simulation. I use the ILA to get signal traces out of the
FPGA.

Regarding the memory addresses, i assigned every address to be the same as in
the ISE/XPS version of ReconOS. In the beginning, i did it wrong and the
system crashed at the reconos_init() call, since it accessed invalid
addresses. Now, the initialization works, so i assume all addresses are correct.

Additionally, the request i see is for (physical) address 0x1dc00040. This is
inside the range of the DDR SDRAM controller, which is 0x0 and 0x1fffffff.

>> Do you have any suggestion on how to solve or investigate into this problem?
> If you are running on Linux, I think you can print the L1 and L2 page table
> addresses (using "virt_to_phys" for L1 address, can check
> "case RECONOS_PROC_CONTROL_SET_PGD_ADDR" in "proc_control_ioctl()" function

> fromlinux/driver/proc_control.c) and see whether it matches your simulation.

I implemented a __printk in that function and had a look at the addresses in
use. I ran sortdemo several times and these are the page table addresses given
to the reconos_zynq_mmu:

0x1de44000
0x1e538000
0x1e538000
0x1de2c000
0x1e630000
0x1e630000
0x1de2c000

What i see via ILA in the runtime simulation are these addresses with an
offset of 0x40 added, so for example: 0x1de2c000 -> 0x1de2c040.

The address asked for by the hardware thread is always 0x000a45e8.
I had a look at the MMU implementation, but i don't understand where this 0x40
offset comes from:

l1_table_addr <= MMU_Pgd(31 downto 14);
l1_descr_addr <= l1_table_addr(31 downto 14) & mem_addr(31 downto 20) & "00";


Address bits 31 downto 14 of l1_table_addr, which is 0x1de2c000 for example, are:
0x1de2c

Address bits 31 downto 20 of the requested virtual address 0x000a45e8 are:
0x000

Together l1_descr_addr forms to 0x1de2c & 0x000 & "00".
So, where does the 0x40 come from?



Kind regards
Sebastian

[Guanwen (Henry) Zhong]

Hi Sebastian,

> Together l1_descr_addr forms to 0x1de2c  & 0x000 & "00".  So, where does the 0x40 come from? 

That looks strange. Can you get the input/output signals for "reconos_memif_mmu_zynq" and post the waveform? It would be easier to debug.

Thanks,

Guanwen

[Sebastian Meisner]

Hi Folks!

I followed another path of investigation and concluded that i have an
interrupt problem somewhere...

The MMU does receive data from the page table and encounters a page miss, so
it goes into the STATE_PAGE_FAULT state and sets signal MMU_Pgf to '1' . Then,
proc_control recognizes this and again sets PROC_Pgf_Int to '1'. Now this is
an interrupt line connected to the CPU, and via an ILA i can see in the
runtime simulation that it is '1'. However, it seems that the driver does not
handle this interrupt. The MMU never gets a MMU_retry set to '1'.

You find my device tree file attached to this email. The problem has to lie
in the hardware somewhere since a design generated with ISE/XPS works fine
with the very same software setup. I have connected PROC_Pgf_Int to the most
significant bit of IRQ_F2P of the Zynq processing system. This should make it
connect to interrupt 91. See design_1.pdf for details.

The Debug output from the reconos drivers is:

mreconos: loading out-of-tree module taints kernel.
[reconos] initializing driver ...
[reconos-proc-control-STATIC] found device reconos_proc_control
[reconos-proc-control-STATIC] found memory at 0x6fe00000 with size 0x10000
[reconos] detected 2 HWTs
[reconos-proc-control] initializing driver ...
[reconos-proc-control] found device reconos_proc_control
[reconos-proc-control] found memory at 0x6fe00000 with size 0x10000
[reconos-proc-control] found interrupt 165
[reconos-proc-control] driver initialized successfully
[reconos-osif] initializing driver ...
[reconos-osif] found device reconos_osif_intc
[reconos-osif] found memory at 0x7b400000 with size 0x10000
[reconos-osif] found interrupt 164
[reconos-osif-intc] registered interrupt controller
random: crng init done
[reconos-proc-control] set_pgd_address: 0x1e694000
[reconos-osif-intc] waiting for interrupt 0
[reconos-osif-intc] enabling interrupt 0
[reconos-proc-control] interrupted, aborting ...
[reconos-osif-intc] interrupted in waiting, aborting ...
[reconos-osif-intc] disabling interrupt 0


cat /proc/interrupts outputs:

CPU0 CPU1
16: 0 0 GIC-0 27 Edge gt
17: 0 0 GIC-0 43 Level ttc_clockevent
18: 437493 13036 GIC-0 29 Edge twd
21: 43 0 GIC-0 39 Level f8007100.adc
142: 0 0 GIC-0 35 Level f800c000.ocmc
143: 426 0 GIC-0 82 Level xuartps
144: 0 0 GIC-0 51 Level e000d000.spi
145: 23473 0 GIC-0 54 Level eth0
146: 0 0 GIC-0 56 Level mmc0
147: 0 0 GIC-0 45 Level f8003000.dmac
148: 0 0 GIC-0 46 Level f8003000.dmac
149: 0 0 GIC-0 47 Level f8003000.dmac
150: 0 0 GIC-0 48 Level f8003000.dmac
151: 0 0 GIC-0 49 Level f8003000.dmac
152: 0 0 GIC-0 72 Level f8003000.dmac
153: 0 0 GIC-0 73 Level f8003000.dmac
154: 0 0 GIC-0 74 Level f8003000.dmac
155: 0 0 GIC-0 75 Level f8003000.dmac
156: 0 0 GIC-0 40 Level f8007000.devcfg
163: 0 0 GIC-0 41 Edge f8005000.watchdog
164: 0 0 GIC-0 90 Level reconos-osif-intc
165: 0 0 GIC-0 91 Level reconos-proc-control
IPI1: 0 0 Timer broadcast interrupts
IPI2: 1624 2896 Rescheduling interrupts
IPI3: 26 24 Function call interrupts
IPI4: 0 0 CPU stop interrupts
IPI5: 0 0 IRQ work interrupts
IPI6: 0 0 completion interrupts
Err: 0



Anyone any ideas?

Kind regards
Sebastian

> --
> You received this message because you are subscribed to the Google Groups
> "ReconOS" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to reconos+u...@googlegroups.com
> <mailto:reconos+u...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[chri...@muppetnet.net]

Hi Sebastian,

 

maybe the IDs of the interrupts may be incorrect. Take a look into answer record 58943 (http://www.xilinx.com/support/answers/58942.html). It describes to use a concat block as you are doing, but at the very end notes, that the input ordering change in latest Vivado versions. So maybe the two interrupts are not connected to 15 (91) and 14 (90) but to interrupt lines 0 (61) and 1 (62). You should be able to check that by simply modifying the device tree for the different interrupt lines.

 

Yours,

Christoph

To unsubscribe from this group and stop receiving emails from it, send an email to reconos+u...@googlegroups.com.

[Sebastian Meisner]

Hi folks!

Thanks to Christoph! The assignment of IRQs was wrong. I had to widen the
concat module to a width of 16 and assign the interrupts to port 14 and 15.
This way we don't have to change the software configuration when migrating
from ISE/XPS to Vivado....

Now interrupts work! However i'm getting funny page fault addresses:

/opt/reconos # ./sortdemo 1 0 16
Initializing ReconOS subsystem ...
[reconos-osif-0] opening ...
[[reconos-proc-control] set_pgd_address: 0x1de8c000
reconos-osif-1] opening ...
creating 1 hw-threads: 0[reconos-osif[reconos-proc-control] page fault occured
-0] writing 0x1000000 ...
[reconos_core] page fault occured at address 1000000
Segmentation fault
/opt/reconos #


I'll have to investigate further, but at least i'm making progress!

Kind regards,
Sebastian

On 16.03.2017 21:37, chri...@muppetnet.net wrote:
> Hi Sebastian,
>
>
>
> maybe the IDs of the interrupts may be incorrect. Take a look into answer
> record 58943 (http://www.xilinx.com/support/answers/58942.html). It describes
> to use a concat block as you are doing, but at the very end notes, that the
> input ordering change in latest Vivado versions. So maybe the two interrupts
> are not connected to 15 (91) and 14 (90) but to interrupt lines 0 (61) and 1
> (62). You should be able to check that by simply modifying the device tree for
> the different interrupt lines.
>
>
>
> Yours,
>
> Christoph
>
>
>

> *Von: *Sebastian Meisner <mailto:sebastia...@upb.de>
> *Gesendet: *Donnerstag, 16. März 2017 17:31
> *An: *rec...@googlegroups.com <mailto:rec...@googlegroups.com>
> *Betreff: *Re: [reconos] ReconOS and Vivado