memcpy over NTB

[Glimcher, Boris]

Hi,

 

I’ve been testing NTB over Microsemi PCIe switch for a few weeks using NTB_PERF

and other tools from the kernel over ntb_hw_switchtec.c (from https://github.com/Microsemi/switchtec-kernel)

 

I’d like to move to user space now and for starters I would like to do a simple MEMCPY over NTB to the peer’s memory.

 

What the best approach for this? What kernel drivers I must use/modify?

 

Thanks,

--

Best regards, Boris

[Logan Gunthorpe]

On 5/1/2018 6:09 AM, Glimcher, Boris wrote:
> I’d like to move to user space now and for starters I would like to do a
> simple MEMCPY over NTB to the peer’s memory.

This isn't supported by the kernel at all. It also introduces
significant security concerns. I'm afraid you're on your own if you want
to do something like that.

I'd suggest developing a custom in-kernel NTB client to suit your needs
instead.

Logan

[Dave Jiang]

There are a lot of people that create VFIO user drivers in place of the kernel driver. That's probably an approach. Also, VFIO mediated device approach may also be a possibility. Have the kernel driver configure and carve out a virtual device for your app to consume.

 

Thanks,

--

Best regards, Boris

--
You received this message because you are subscribed to the Google Groups "linux-ntb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to linux-ntb+...@googlegroups.com.
To post to this group, send email to linu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/linux-ntb/8A1F3ACA2F40C04FBF91985D6F491D7C017448F2%40MX302CL01.corp.emc.com.
For more options, visit https://groups.google.com/d/optout.

[Glimcher, Boris]

So one approach could be to modify kernel driver and use ioctl from user space to submit buffers for transfer and kernel driver will copy or use dma to copy the buffer to the other side?
What do you think? Valid?

--
Best regards, Boris
Let me google that for you

[Logan Gunthorpe]

On 5/1/2018 10:09 AM, Glimcher, Boris wrote:
> So one approach could be to modify kernel driver and use ioctl from user space to submit buffers for transfer and kernel driver will copy or use dma to copy the buffer to the other side?
> What do you think? Valid?

Yes, I think that'd be a good choice. Create an in-kernel NTB client
with a char device which has an IOCTL or read/write interface for
submitting buffers. This keeps MW setup and DMA operations inside the
kernel.

Logan

[Glimcher, Boris]

Is it also possible with this in-kernel NTB client with a char device to MMAP somehow MW to user?

So the kernel code will be limited just to setup the connection section and will not take part during data transfer at all?
Cause I see that eventually kernel driver just does "memcpy_toio"...

Am I missing anything?

--
Best regards, Boris
Let me google that for you


-----Original Message-----
From: Logan Gunthorpe [mailto:log...@deltatee.com]
Sent: Tuesday, May 1, 2018 19:15
To: Glimcher, Boris; linu...@googlegroups.com
Subject: Re: memcpy over NTB

[Logan Gunthorpe]

On 5/1/2018 10:20 AM, Glimcher, Boris wrote:
> Is it also possible with this in-kernel NTB client with a char device to MMAP somehow MW to user?

Yes, this is possible but generally discouraged for drivers.

If you are attempting to optimize for performance then it's much better
to submit large buffers to a DMA engine.

If you are trying to have random access to the MW you'll have to be very
careful about synchronization with the opposite side.

Logan

[Glimcher, Boris]

The synchronization is not an issue, I believe, as I'm looking just to mirror large memory area from one side to another

--
Best regards, Boris
Let me google that for you


-----Original Message-----
From: Logan Gunthorpe [mailto:log...@deltatee.com]
Sent: Tuesday, May 1, 2018 19:26
To: Glimcher, Boris; linu...@googlegroups.com
Subject: Re: memcpy over NTB

[Logan Gunthorpe]

On 5/2/2018 2:06 AM, Glimcher, Boris wrote:
> The synchronization is not an issue, I believe, as I'm looking just to mirror large memory area from one side to another

What happens if one side is reading while the other side is changing the
memory? They'd read some of the data before the change and some of the
data after the change resulting in inconsistent data. "just mirroring"
doesn't solve those issues.

Logan

[Glimcher, Boris]

Invalid situation. One side is always writing. Other side is never reading until first side is down. (High Availability stuff)

--
Best regards, Boris
Let me google that for you


-----Original Message-----
From: linu...@googlegroups.com [mailto:linu...@googlegroups.com] On Behalf Of Logan Gunthorpe
Sent: Wednesday, May 2, 2018 18:43
To: Glimcher, Boris; linu...@googlegroups.com
Subject: Re: memcpy over NTB

--
You received this message because you are subscribed to the Google Groups "linux-ntb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to linux-ntb+...@googlegroups.com.
To post to this group, send email to linu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/linux-ntb/4f05cc88-5ad6-e8f0-070c-0cc62d30d878%40deltatee.com.