Misleading throughput number using ntb_perf with memcpy_toio()

[Karl Kao]

Hi Folks,

We have one system with PLX NTB enabled with Gen3 speed on x8 lanes width. Technically, the NTB bandwidth is 64 Gbit/s (8192 MBytes/s) in outbound direction to its peer on this system.

The ntb_perf comes with the throughput number of 9147 MBytes/s which is misleading in comparison to hardware bandwidth of 8192 MBytes/s.

We have a few questions with regard to the misleading throughput number. The overall is when memcpy() returns, data may have not yet been put in ingress buffer inside PLX NTB which is constrained by flow control credits, limitation of 8192 Mbytes/s.

• The ntb_perf moves memory block to IO bus, using memcpy_toio(), a macro of memcpy(). Once cpu core is done with mov instructions, the memcpy() returns. Is this statement accurate?
  
• At the point when memcpy() is done with mov instructions, would the data had been moved to internal buffer inside processor, instead of going through IO bus into PLX chip?
• Where is the buffer, IIO buffer? What's buffer size that can accommodate gigabytes data?
• Once we are clearer about the memcpy_tpio, would there be any API that can accurately measure the IO write throughput?

[11:02][root@BRYCE2-DEV][/sys/kernel/debug/ntb_perf/0000:06:00.0]\> cat run

0: copied 4294967296 bytes in 469542 usecs, 9147 MBytes/s

[11:03][root@BRYCE2-DEV][/sys/kernel/debug/ntb_perf/0000:06:00.0]\>

Thanks,

Karl

[Dave Jiang]

On 11/02/2017 11:33 AM, Karl Kao wrote:
> Hi Folks,
>
> We have one system with PLX NTB enabled with Gen3 speed on x8 lanes
> width. Technically, the NTB bandwidth is 64 Gbit/s (8192 MBytes/s) in
> outbound direction to its peer on this system.
> The ntb_perf comes with the throughput number of 9147 MBytes/s which is
> misleading in comparison to hardware bandwidth of 8192 MBytes/s.
>
> We have a few questions with regard to the misleading throughput number.
> The overall is when memcpy() returns, data may have not yet been put in
> ingress buffer inside PLX NTB which is constrained by flow control
> credits, limitation of 8192 Mbytes/s.
>

> * The ntb_perf moves memory block to IO bus, using memcpy_toio(), a

> macro of memcpy(). Once cpu core is done with mov instructions, the
> memcpy() returns. Is this statement accurate?

> * At the point when memcpy() is done with mov instructions, would the

> data had been moved to internal buffer inside processor, instead of
> going through IO bus into PLX chip?

> * Where is the buffer, IIO buffer? What's buffer size that can
> accommodate gigabytes data?
> * Once we are clearer about the memcpy_tpio, would there be any API

> that can accurately measure the IO write throughput?
>
>
> [11:02][root@BRYCE2-DEV][/sys/kernel/debug/ntb_perf/0000:06:00.0]\> cat run
> 0: copied 4294967296 bytes in 469542 usecs, 9147 MBytes/s
> [11:03][root@BRYCE2-DEV][/sys/kernel/debug/ntb_perf/0000:06:00.0]\>

Hi Karl. Yes the issue with ntb_perf is that it blindly copies into the
memory window with no acknowledgement from the peer node. So the
performance can be somewhat misleading because you don't know if it has
made it completely to the other side. The only way to know for certain
everything has made it over is to do a small read at the end of the
write I think?

[Allen Hubbe]

From: Dave Jiang

Or, a warm-up phase to fill up write queues, before the measured phase of the throughput test?

We don't need to know that it made it to the other side. That would be needed for latency, but for throughput we only care about the rate it was transmitted even if it takes a long time to arrive.

[Karl Kao]

Hi Dave, the no acknowledge from peer is justified by doing posted pcie transactions. I don’t understand why memcpy can return and claim it’s done the job with assumption that internal buffer is sufficient to take gigabytes data.

[Dave Jiang]

On 11/02/2017 11:48 AM, Karl Kao wrote:
> Hi Dave, the no acknowledge from peer is justified by doing posted pcie
> transactions. I don’t understand why memcpy can return and claim it’s
> done the job with assumption that internal buffer is sufficient to take
> gigabytes data.

memcpy is just CPU instructions right? It returns as soon as CPU is done
copy. I don't think it has any ideas with regards to I/O buffers or
anything like that.

>
>  
>
>  
>
>  
>
> *From: *Dave Jiang <dave....@intel.com>
> *Date: *Thursday, November 2, 2017 at 11:39
> *To: *Karl Kao <yk...@tintri.com>, linux-ntb <linu...@googlegroups.com>
> *Subject: *Re: Misleading throughput number using ntb_perf with

> --
> You received this message because you are subscribed to the Google
> Groups "linux-ntb" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to linux-ntb+...@googlegroups.com
> <mailto:linux-ntb+...@googlegroups.com>.
> To post to this group, send email to linu...@googlegroups.com
> <mailto:linu...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/linux-ntb/56307FCF-28EE-4857-BA37-86D84CD617F7%40tintri.com
> <https://groups.google.com/d/msgid/linux-ntb/56307FCF-28EE-4857-BA37-86D84CD617F7%40tintri.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Karl Kao]

Yes, the memcpy is a set of CPU instructions. However, CPU is moving data from regular memory to destination address targeted at I/O. Wouldn’t the CPU move data to certain I/O buffer before the I/O bus is free to take?