ReconOS and Vivado - Caching Problems?

[Sebastian Meisner]

Hi folks!

I'm still trying to get ReconOS to run with Vivado. Currently, i experience
problems when accessing main memory via MEMIF. I checked the MMU, reconos
memory controller , AXI4 bus at memory controller side and finally AXI3 bus at
the Zynq AXI ACP port.

The attached image shows the signals of one bus transaction at the Zynq ACP
port when trying to read the sort data in the sortdemo. The reconos memory
controller requests 256 bytes from memory and the AXI3 bus at the ACP port
gets four 64 byte requests. However, if you look at the data signals (colored
in red), you see that most of the time they are all zeros, but suddenly, in
the middle of the transaction, 32 bytes get transferred.

My conclusion so far is that all busses work as expected, but we have a
problem with caching. The above mentioned 32 bytes are exactly the length of
an ARM cache line. It looks like, whenever data gets transferred its a
multiple of 32 bytes. And what data gets transferred is also
non-deterministic, which supports the cache theory.

Long story short: Does anybody know how to switch of cache in the processing
system? Or at least if something cache related was changed when switching from
XPS to Vivado?

Kind regards,
Sebastian

[Christoph Rüthing]

Hi Sebastian,

when developing the current master controller for the ACP of the Zynq I also struggled some caching issues and I have some ideas/suggestions you could investigate:

• The bus transactions themselves look okay in my opinion. Originally, ReconOS eventually splits up big requests into smaller ones of 64 words (32bit) and I am not sure how these are translated by you to bus transactions for the ACP (maybe these 64 word request (256 byte) are, again, split up into smaller ones).
• When I encountered caching issues earlier, I also had them for the page tables, but I believe you have checked the addresses requested on the bus, since you already posted some issues with the address translation.?
• The ACP has special signals to select between different modes of cache coherency for a request via the _ARCACHE and ARUSER signals. At least in XPS, these signals were not connected automatically (needed to add them manually to mpd, don't know what Vivado is doing). Unfortunately, they are rather undocumented in the Zynq TRM provided by Xilinx, but there is some documentation from ARM.
• Zynq TRM states in section 3.5.1 that at least _ARUSER[0] and _ARCACHE[1] needs to be set to one.
• http://infocenter.arm.com/help/topic/com.arm.doc.ddi0246h/ch02s03s01.html provides descriptions for _ARCACHE signals.
• http://infocenter.arm.com/help/topic/com.arm.doc.ddi0407i/CIAGAAJH.html provides descriptions for _ARUSER signals.
• I see in the trace, that you already use the _ARCACHE (0x3) signals, but do not have the _ARUSER signals included. For the current ReconOS memory controller I set both signals to all ones (see https://github.com/ReconOS/reconos/blob/develop/lib/pcores/reconos_memif_memory_controller_v1_00_a/hdl/vhdl/reconos_memif_memory_controller.vhd#L150) as done by many other users and it worked fine. If needed, you can also tweak this signals to your needs, but make sure to fulfill the requirements for cache-coherent access.

Maybe that helps you, it took some time to find that out when I was developing the ACP master component.

Yours,
Christoph

[Sebastian Meisner]

Hi Christoph!

Thanks for your answer. And just to clarify: i am using your reconos memory
controller implementation. :-)

- I checked the splitting of burst transactions and they look fine, so this
seems not to be the issue.

- The caching issues also seem to relate to the page table, although they do
occur less frequent.

- Regarding the ARCACHE amd ARUSER signals: This seems currently to be the
problem. I have trouble verifying that they have the correct value at the ACP
port. Due to the protocol conversion (AXI4 to AXI3) and bus width adaptation
the exact wiring is intransparent to me. I tried to make additional direct
connections between M_AXI_ARUSER and S_AXI_ARUSER but i'm not sure if Vivado
wires it accordingly.

Additionally, the signals that Vivado shows me in the block design view differ
from the ones i get via ILA. In the Block Design View the ACP port has
ARUSER[4:0] signals, but in the ILA i can't find them at all. Thus, i can't
check their actual values. This is really strange...


Thanks again for your answer, now i know i'm searching at the right places.

Kind regards,
Sebastian

On 29.03.2017 12:10, Christoph Rüthing wrote:
> Hi Sebastian,
>
> when developing the current master controller for the ACP of the Zynq I also
> struggled some caching issues and I have some ideas/suggestions you could
> investigate:
>

> * The bus transactions themselves look okay in my opinion. Originally,

> ReconOS
eventually splits up big requests into smaller ones of 64 words
> (32bit) and I am not sure how these are translated by you to bus
> transactions for the ACP (maybe these 64 word request (256 byte) are,
> again, split up into smaller ones).

> * When I encountered caching issues earlier, I also had them for the page

> tables, but I believe you have checked the addresses requested on the bus,
> since you already posted some issues with the address translation.?

> * The ACP has special signals to select between different modes of cache

> coherency for a request via the _ARCACHE and ARUSER signals. At least in
> XPS, these signals were not connected automatically (needed to add them
> manually to mpd, don't know what Vivado is doing). Unfortunately, they are
> rather undocumented in the Zynq TRM provided by Xilinx, but there is some
> documentation from ARM.

> o Zynq TRM states in section 3.5.1 that at least _ARUSER[0] and

> _ARCACHE[1] needs to be set to one.

> o http://infocenter.arm.com/help/topic/com.arm.doc.ddi0246h/ch02s03s01.html

> provides descriptions for _ARCACHE signals.

> o http://infocenter.arm.com/help/topic/com.arm.doc.ddi0407i/CIAGAAJH.html provides
> descriptions for _ARUSER signals.
> * I see in the trace, that you already use the _ARCACHE (0x3) signals, but

> do not have the _ARUSER signals included. For the current ReconOS memory
> controller I set both signals to all ones (see
> https://github.com/ReconOS/reconos/blob/develop/lib/pcores/reconos_memif_memory_controller_v1_00_a/hdl/vhdl/reconos_memif_memory_controller.vhd#L150)
> as done by many other users and it worked fine. If needed, you can also
> tweak this signals to your needs, but make sure to fulfill the
> requirements for cache-coherent access.
>
> Maybe that helps you, it took some time to find that out when I was developing
> the ACP master component.
>
> Yours,
> Christoph
>

> --
> You received this message because you are subscribed to the Google Groups
> "ReconOS" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to reconos+u...@googlegroups.com
> <mailto:reconos+u...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Christoph Rüthing]

Aha, interesting, didn't know that the axi_master_burst IP-Core is also available in Vivado. I thought you have rewritten the core without this intermediate core.

I just configured a Zynq PS in Vivado and found the "Tie off AxUSER" setting, maybe that helps.

Yours,
Christoph

[Sebastian Meisner]

Hi Folks!

Okay i just found this:

https://www.xilinx.com/support/answers/62051.html

The AXI data width converter does not pass through the AxUSER signals. This
may be the reason why i don't get sensible data out of the ACP.

So, either we somehow force the AxUSER signals somehow to be all '1' or we
extend the memory controller to be a 64 bit bus master instead of a 32 bit
one. Any opinions on that?

@Christoph: I already checked the "Tie off AxUSER" setting, but it did not
help :-(

Kind regards
Sebastian

[Sebastian Meisner]

On 29.03.2017 16:42, Sebastian Meisner wrote:
> @Christoph: I already checked the "Tie off AxUSER" setting, but it did not
> help :-(

I withdraw my last statement! After i build a clean ReconOS system in Vivado
with my TCL script and checked the "Tie off AxUSER" the sortdemo worked!

We have a working bitstream out of Vivado! Yeah! :-D

Kind regards
Sebastian

[Anon A]

HI Sebastian,

Is it possible for you to post a tutorial on buliding ReconOS with Vivado toolchain. I am a beginner both with Vivado and ReconOS. it would be very helpful if you can post the tutorial on  setting up reconOS with vivado toolchain.

Thanks,
Anon

[Sebastian Meisner]

Hi Anon!

Unfortunately, a tutorial will be not enough to build ReconOS with the Vivado
toolchain. Quite some extensions to our ReconOS Development Kit and a lot of
TCL scripts are needed get Vivado and ReconOS work together.

This work is still ongoing, but we plan to release it in some weeks. Please be
patient with us!

Kind regards,
Sebastian Meisner

> --
> You received this message because you are subscribed to the Google Groups
> "ReconOS" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to reconos+u...@googlegroups.com
> <mailto:reconos+u...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

The