Make BRAM/DRAM from FPGA accessible to Jailhouse on ZCU102

[Giovani Gracioli]

Hello,

I have a BRAM and DRAM blocks available on the FPGA side of the ZCU102 platform.

I would like to make them accessible for jailhouse cells. For instance, a BRAM block is mapped to the 0xa0000000 address.

To do that, I inserted this address into the root and non-root cell configurations as mem regions:

Non-root:

{
.phys_start = 0x00A0000000,
.virt_start = 0x00A0000000,
.size = 0x200000, //2MB
.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE
},

Root:

{
.phys_start = 0x00A0000000,
.virt_start = 0x00A0000000,
.size = 0x200000, //2MB
.flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_EXECUTE,
},

When I run the non-root cell and try to write into the address:

int *bram_ptr = (int *) 0x00A0000000;
*bram_ptr = 10;

I got an unhandled data write at 0xa0000000. What am I missing here in the configs?

Thanks
Giovani

[Jan Kiszka]

Maybe the region was not properly registered. Did you increase the
mem_regions array size? Did you change num_memory_regions so that it
requires manual updates? That's better discussed over the full config.

Note that, if you want to share a region between root and non-root cell,
the latter also needs JAILHOUSE_MEM_ROOTSHARED.

Jan

[Giovani Gracioli]

Hi,

I checked the array sizes and they are correct. num_memory_regions uses the ARRAY_SIZE macro:

.num_memory_regions = ARRAY_SIZE(config.mem_regions)

In Linux I can mmap and access the mapped region without errors.

Any other thought?

[Giovani Gracioli]

Don't know exactly why, but if I change the size for 1MB instead of 2MB, it works.

Is this because of this line in the mmu_cell.c?

size = MIN(region_size, NUM_TEMPORARY_PAGES * PAGE_SIZE);

Thus, is the maximum size of any memory region 1MB?

[Jan Kiszka]

On 2018-05-23 20:53, Giovani Gracioli wrote:
> Don't know exactly why, but if I change the size for 1MB instead of 2MB, it works.
>
> Is this because of this line in the mmu_cell.c?
>
> size = MIN(region_size, NUM_TEMPORARY_PAGES * PAGE_SIZE);
>
> Thus, is the maximum size of any memory region 1MB?
>

No, the minimum size is 4K on all supported architectures.

Without the configs you were using, it's hard to say where the mistake is.

Also, what kind of inmate are you booting in the non-root cell?

Jan

>> Hi,
>>
>> I checked the array sizes and they are correct. num_memory_regions uses the ARRAY_SIZE macro:
>>
>> .num_memory_regions = ARRAY_SIZE(config.mem_regions)
>>
>> In Linux I can mmap and access the mapped region without errors.
>>
>> Any other thought?
>>
>>>
>>> Maybe the region was not properly registered. Did you increase the
>>> mem_regions array size? Did you change num_memory_regions so that it
>>> requires manual updates? That's better discussed over the full config.
>>>
>>> Note that, if you want to share a region between root and non-root cell,
>>> the latter also needs JAILHOUSE_MEM_ROOTSHARED.
>>>
>>> Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[Giovani Gracioli]

Hi,

The configs are attached. I am running Erika on the non-root cell.

[Jan Kiszka]

On 2018-05-24 15:44, Giovani Gracioli wrote:
> Hi,
>
> The configs are attached. I am running Erika on the non-root cell.

Does Erika enable the MMU? Is it prepared to map memory from higher
addresses?

I'm now seeing in your configs that you map a region at 0x500000000 - Is
the access fault reported for that address as well?

[Giovani Gracioli]

Not sure about MMU on Erika.

Yes, the fault is reported as well. This address is just another memory. 0xA00.. is the BRAM and 0x500.. is the DRAM. Both have the same faulty behavior when the size is 2MB.

[Jan Kiszka]

On 2018-05-24 15:59, Giovani Gracioli wrote:
> Not sure about MMU on Erika.
>
> Yes, the fault is reported as well. This address is just another memory. 0xA00.. is the BRAM and 0x500.. is the DRAM. Both have the same faulty behavior when the size is 2MB.

You see the issue also when mapping some arbitrary DRAM region with a
size of 2MB? Can you reproduce this with an IVSHMEM region as well? What
if you increase the size beyond 2M, to 3M e.g.?

[Giovani Gracioli]

DRAM: works with 1MB, 3MB, and 32MB.
does not work with 2MB.

BRAM (is limited to 2MB): works with 1MB, does not with 2MB.

IVSHMEM: tested with 1 and 2MB and worked in both cases.

[Jan Kiszka]

That's not a consistent pattern yet because IVSHMEM and DRAM is just the
same, use the same flags etc.

Can you send a config that demonstrates the DRAM issue? Ideally very
close to the upstream zynqmp-zcu102 configs.

Thanks,

[Giovani Gracioli]

Just tested these two attached and got:

Unhandled data write at 0x500000000(4)

FATAL: unhandled trap (exception class 0x24)
Cell state before exception:
pc: 0000000000006944 lr: 0000000000006944 spsr: 60000005 EL1
sp: 0000000000006870 esr: 24 1 1970045
x0: 0000000000000000 x1: 0000000000009670 x2: 0000000000000000
x3: 0000000000000004 x4: 00000000000027b8 x5: 0000000000000000
x6: 0000000000000000 x7: 0000000000000080 x8: 00000000000068b0
x9: 00000000000023e0 x10: 0000000000000000 x11: 0000000000009292
x12: 0000000000000000 x13: 0000000000000000 x14: 0000000000000020
x15: 0000000000000000 x16: 0000000000000000 x17: 0000000000000000
x18: 0000000000000000 x19: 0000000000001118 x20: 00000000000092cf
x21: 0000000800700000 x22: 0000000500000000 x23: 000000000000000a
x24: 0000000000000000 x25: 0000000000000000 x26: 0000000000000000
x27: 0000000000000000 x28: 0000000000000000 x29: 0000000000000000

Parking CPU 1 (Cell: "nfer")

[Jan Kiszka]

OK, good news, I've reproduced something very strange, and that even
inside qemu. Debugging...

[Giovani Gracioli]

Let me know if you want me to test on the real hardware once you have figure it out.