Virtio driver provides wrong address in the virtqueue descriptor table

[super...@gmail.com]

I'm running Linux host and FreeRTOS remote on a Zynq7000 system.
Total memory is 1 GiB. The upper 256 MiB (0x30000000-0x40000000) is reserved for the remote core.
In the 256 MiB region, the first 128 MiB is reserved for the ELF loading. This leaves the upper 128 MiB (0x38000000-0x40000000) available for shared memory between the cores.

I'm using regular RPMsg transport, and I want to extend the functionality by adding a Virtio Network driver.
I've extended the resource table with a new network vdev and two additional vrings.
When starting the remote core Linux and remote core, Linux loads the VIRTIO_NET driver and correctly negotiate features for the new vdev.

My problem is that when I send network data to the remote core, the addr field in the struct vring_desc is in Linux memory area (0x2e......).
The addr field for Rpmsg vqueue descriptors are correctly located at 0x3804.... and 0x3806....

I've noticed that you can set the da address for vrings in the resource table (rpmsg example use RING_TX/RING_RX), but changing this does not seem to have any effect at all. I've also tried setting all vring addresses to FW_RSC_U32_ADDR_ANY, but no change.

How can I make Linux set the correct address in virtqueue descriptors for the new vrings?

Gaute

[Ben Levinsky]

Just to be clear, are you using RPMsg in kernel space or userspace? Also can you provide your resource table and device tree?


Best,
Ben

--
You received this message because you are subscribed to the Google Groups "open-amp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to open-amp+u...@googlegroups.com.
To post to this group, send an email to open...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/open-amp/3d9b901c-2d0a-428c-aebd-98e1183dc0d3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[super...@gmail.com]

I'm using RPMsg in userspace from the proxy_app example, but I don't see how that matters? RPMsg should be a separate virtio class.

Resource table:

#define NO_RESOURCE_ENTRIES         8

METAL_PACKED_BEGIN
struct virtio_net_config {
    uint8_t mac[6];
//  uint16_t status;
//  uint16_t max_virtqueue_pairs;
//  uint16_t mtu;
} METAL_PACKED_END;

/* Resource table for the given remote */
struct remote_resource_table {
    unsigned int version;
    unsigned int num;
    unsigned int reserved[2];
    unsigned int offset[NO_RESOURCE_ENTRIES];

    /* rpmsg vdev entry */
    struct fw_rsc_vdev rpmsg_vdev;
    struct fw_rsc_vdev_vring rpmsg_vring0;
    struct fw_rsc_vdev_vring rpmsg_vring1;

    /* net vdev entry */
    struct fw_rsc_vdev net_vdev;
    struct fw_rsc_vdev_vring net_vring0;
    struct fw_rsc_vdev_vring net_vring1;
    struct virtio_net_config net_cfg;
}__attribute__((packed, aligned(0x100000)));

#define RING_TX                     FW_RSC_U32_ADDR_ANY  // it does not matter what I put here...

#define RING_RX                     FW_RSC_U32_ADDR_ANY

#define NET_RING_TX                 FW_RSC_U32_ADDR_ANY

#define NET_RING_RX                 FW_RSC_U32_ADDR_ANY

#define VRING_SIZE                  64

#define NUM_TABLE_ENTRIES           2

struct remote_resource_table __resource resources = {
    .version = 1,  // version number
    .num = NUM_TABLE_ENTRIES,  // number of table entries
    .reserved = {0, 0,},  // reserved (must be zero)
    .offset = { // Offsets of rsc entries
        offsetof(struct remote_resource_table, rpmsg_vdev),
        offsetof(struct remote_resource_table, net_vdev),
    },
    .rpmsg_vdev = {
        RSC_VDEV, VIRTIO_ID_RPMSG_, 0,
        RPMSG_IPU_C0_FEATURES, 0, 0, 0, NUM_VRINGS, {0, 0},
    },
    .rpmsg_vring0 = { RING_TX, VRING_ALIGN, VRING_SIZE, 0, 0 },
    .rpmsg_vring1 = { RING_RX, VRING_ALIGN, VRING_SIZE, 1, 0 },

    .net_vdev = {
       RSC_VDEV, VIRTIO_ID_NET_, 1,
       VIRTIO_NET_FEATURES, 0, sizeof(struct virtio_net_config), 0, NUM_VRINGS, {0, 0},
    },
    .net_vring0 = { NET_RING_TX, VRING_ALIGN, VRING_SIZE, 2, 0 },
    .net_vring1 = { NET_RING_RX, VRING_ALIGN, VRING_SIZE, 3, 0 },
    .net_cfg = {{1, 2, 3, 4, 5, 6}, /*VIRTIO_NET_S_LINK_UP*/},
};

Relevant parts of device tree:

reserved-memory {
    #address-cells = <1>;
    #size-cells = <1>;
    ranges;
    rproc_0_reserved: rproc@30000000 {
       no-map;
       reg = <0x30000000 0xff00000>; /* start address: 768 MiB, length: 255 MiB */
    };
};

amba {
    elf_ddr_0: ddr@30000000 {
        compatible = "mmio-sram";
        reg = <0x30000000 0x7f00000>; /* start address: 768 MiB, length: 127 MiB */
    };
};

remoteproc {
    compatible = "xlnx,zynq_remoteproc";
    vring0 = <15>;
    vring1 = <14>;
    srams =  <&elf_ddr_0>;
};

Gaute

On Monday, August 5, 2019 at 10:47:16 PM UTC+2, Ben Levinsky wrote:

Just to be clear, are you using RPMsg in kernel space or userspace?  Also can you provide your resource table and device tree?


Best,
Ben

-----Original Message-----
From: open...@googlegroups.com <open...@googlegroups.com> On Behalf Of super...@gmail.com
Sent: Tuesday, July 16, 2019 11:57 PM
To: open-amp <open...@googlegroups.com>
Subject: [open-amp] Virtio driver provides wrong address in the virtqueue descriptor table

I'm running Linux host and FreeRTOS remote on a Zynq7000 system.
Total memory is 1 GiB. The upper 256 MiB (0x30000000-0x40000000) is reserved for the remote core.
In the 256 MiB region, the first 128 MiB is reserved for the ELF loading. This leaves the upper 128 MiB (0x38000000-0x40000000) available for shared memory between the cores.

I'm using regular RPMsg transport, and I want to extend the functionality by adding a Virtio Network driver.
I've extended the resource table with a new network vdev and two additional vrings.
When starting the remote core Linux and remote core, Linux loads the VIRTIO_NET driver and correctly negotiate features for the new vdev.

My problem is that when I send network data to the remote core, the addr field in the struct vring_desc is in Linux memory area (0x2e......).
The addr field for Rpmsg vqueue descriptors are correctly located at 0x3804.... and 0x3806....

I've noticed that you can set the da address for vrings in the resource table (rpmsg example use RING_TX/RING_RX), but changing this does not seem to have any effect at all. I've also tried setting all vring addresses to FW_RSC_U32_ADDR_ANY, but no change.

How can I make Linux set the correct address in virtqueue descriptors for the new vrings?


Gaute

--
You received this message because you are subscribed to the Google Groups "open-amp" group.

To unsubscribe from this group and stop receiving emails from it, send an email to open...@googlegroups.com.

[Jiaying Liang]

Hi Gaute,

> -----Original Message-----
> From: open...@googlegroups.com [mailto:open...@googlegroups.com]
> On Behalf Of super...@gmail.com
> Sent: Tuesday, July 16, 2019 11:57 PM
> To: open-amp <open...@googlegroups.com>
> Subject: [open-amp] Virtio driver provides wrong address in the virtqueue
> descriptor table
>

[Wendy] By default, the vring address is allocated by Linux kernel.
Usually, you use predefined vring address with IOMMU.

Did you try to specify DMA memory for your remoteproc driver?

Best Regards,
Wendy

>
> How can I make Linux set the correct address in virtqueue descriptors for the
> new vrings?
>
>
> Gaute
>

> --
> You received this message because you are subscribed to the Google Groups
> "open-amp" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to open-amp+u...@googlegroups.com.

[super...@gmail.com]

I agree that the vring address is allocated by the Linux kernel, but since it allocates it in a wrong area, how can I change this?

I didn't think the Zynq7000 had a IOMMU.

I did not specify DMA memory for the remoteproc driver.

I'm assume this is done from the device tree. How can I do this for the Zynq-7000 system?

Best Regards,

Gaute

> email to open...@googlegroups.com.