Experiments with send in Linux

[John Hening]

Hello,

I am experimenting with send (from socket to socket through a loopback). I create a flamegraph for that and measure time of execution. It seems that is takes relatively much time. (I am not sure whether is it much or not). In test I send a 64-byte message in a loop.

send function takes 10-15 microseconds.

1. Is it much or not?
2. Have you any idea how to improve it? I tried to extend send buffer socket, doesn't help- I know that probably it couldn't help.
3. When it comes to a flamegraph: It looks like the kernel waits for a response- I mean the function ip_recv and so on. What does the kernel try to receive? AFAIK a send function doesn't wait for ACK from receiver.

[Avi Kivity]

On 2018-05-07 01:09, John Hening wrote:

Hello,

I am experimenting with send (from socket to socket through a loopback).

If you want to measure send, don't use loopback. It necessarily includes receiving.

I create a flamegraph for that and measure time of execution. It seems that is takes relatively much time. (I am not sure whether is it much or not). In test I send a 64-byte message in a loop.

send function takes 10-15 microseconds.

1. Is it much or not?

That's a philosophical question that I'll leave to the philosophers.

2. Have you any idea how to improve it? I tried to extend send buffer socket, doesn't help- I know that probably it couldn't help.

Don't use loopback, disable meltdown/spectre mitigation, amortize many operations on one send(), use unix domain sockets or (better) shared memory for IPC.

3. When it comes to a flamegraph: It looks like the kernel waits for a response- I mean the function ip_recv and so on. What does the kernel try to receive? AFAIK a send function doesn't wait for ACK from receiver.

The send doesn't wait, but the kernel has to serve both the send side and the receive side, and if they happen to run on the same CPU and possibly in the same thread, then you'll see them in the flame graph for that CPU or thread.

In this case the kernel decided to run the receive processing immediately, so that's what you get.

[John Hening]

Thanks Avi,
 

The send doesn't wait, but the kernel has to serve both the send side and the receive side, and if they happen to run on the same CPU and possibly in the same thread, then you'll see them in the flame graph for that CPU or thread.

Ok. I didn't notice that I create a flamegraph for a specififc PID, so it is not that case- I mean that I did not sample a CPU but specififc process.

In this case the kernel decided to run the receive processing immediately, so that's what you get.

I see, you right. This is a way kernel implements a loopback.

use unix domain sockets or (better) shared memory for IPC.

I cannot, my communication is not interprocess.

Now everything is clear, thanks ;-)

[Travis Downs]

Don't use loopback, disable meltdown/spectre mitigation, ...

Out of curiosity, have you tried it and how did you do it? I have tried it with spectre_v2=off nopti at kernel boot, but my testing showed this was somewhat slower than the same kernel with mitigations enabled, both of which were much slower than kernels with the mitigations not present (either an older kernel or backing out the patches - the latter option being very difficult on newer kernels with more cumulative changes built on those patches).

The numbers I got (from memory) were something like ~100 cycles of raw syscall overhead before mitigations, and 700 cycles with mitigations, but close to 2000 cycles if your kernel had the mitigations but you disabled them at boot. The numbers we are talking about (less than 1 microsecond) don't take much of a byte out of a 10 - 15 microsecond call, but they matter for short calls for sure.

I saw the same effect when measuring the cost of page faults, which also implies a user-kernel call (albeit indirectly).

[Avi Kivity]

On 2018-06-03 01:33, Travis Downs wrote:

Don't use loopback, disable meltdown/spectre mitigation, ...

Out of curiosity, have you tried it and how did you do it?

I did not try it.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.