Problem with network IO testing with iperf3

[Dmitrii Ustiugov]

Hi everyone,

I am trying to measure network IO performance of gVisor with iperf3. I am booting a docker container with the following cmd:

docker run -dit --name alpine1 --runtime=runsc -p 52022:22 -p 42022:5201 alpine ash

After configuring SSH and few other things, I am able to SSH into the container from the remote machine, no problem. However, when I start an iperf3 server inside that container and try to run the client on a remote host, it just hangs and timeouts.

From the client side I cannot see any output:

$ iperf3 -c 10.90.36.40 -p 42022 -V

iperf 3.0.11

Linux XXXX 4.11.0-13-generic #19~16.04.1-Ubuntu SMP Wed Aug 2 20:06:21 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

From the server side, I see: 

/ # iperf3 -s

-----------------------------------------------------------

Server listening on 5201

-----------------------------------------------------------

iperf3: error - unable to receive parameters from client: Operation timed out

This exact setup works perfectly with the default docker runtime.

I would appreciate any help on this issue. Thank you!

Regards,
Dmitrii

[Fabricio Voznika]

We've noticed a similar problem too and are investigating it. I've seen cases where the client gets connection reset by peer after iperf runs for some time and there are retries. Could you please file a bug in https://github.com/google/gvisor/issues/new with as much details as possible to repro the problem?

Thanks,

Fabricio.

--
You received this message because you are subscribed to the Google Groups "gVisor Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gvisor-users...@googlegroups.com.
To post to this group, send email to gvisor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gvisor-users/ef46af41-fda2-4662-9519-edc1d78173d1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bhasker Hariharan]

I believe the bug is in GSO. If i run runsc with --gso=false the iperf test works fine.

https://github.com/google/gvisor/issues/195#issuecomment-482773906 

I think we are messing up the packet somehow when GSO is on for short writes or something.

-bhasker 

[Bhasker Hariharan]

Bringing the discussion back to the original thread. Answer inline

On Fri, Apr 12, 2019 at 9:18 PM Dmitrii Ustiugov <notifi...@github.com> wrote:

Hi Bhasker, 

Thank you for the updates. I am using port forwarding because I am modeling a Function-as-a-Service setting where micro-VMs are normally behind a NAT. For that reason, I rely on the docker default bridging. 

Could you elaborate a bit more on what GSO is in gvisor? 

GSO is Generic Segmentation Offload which was added recently to the network stack in gVisor. What that does is offloads large segments from the stack to the veth device which can then do the actual segmentation of TCP segments into MTU sized chunks. We do this by prepending a special vnet header when writing the packet to the veth device which causes the host kernel to do the segmentation based on the parameters specified in the header.

I believe we have some bug in the handling of small segments which is causing the iperf hang. When I disable GSO I do not see any hangs with iperf. We will take a look on monday and see if we can get a bug fix in quickly. But for now disabling GSO should help resolve the issue.

-bhasker 

I’ve also noticed the following problem when I used my own super simple TCP server inside a gvisor sandbox. When connections are established, the clients and the server can exchange packets but after my benchmark finishes, if I restart the client and the server, the connections time out and I have to restart the gvisor containers. It must be the same issue, I guess. 

Thank you. 

Regards, 
Dmitrii 

On Apr 13, 2019, at 6:01 AM, Bhasker Hariharan <notifi...@github.com<mailto:notifi...@github.com>> wrote: 


Okay could you create a /etc/docker/daemon.json with the following contents. The critical part is --gso=false. I believe we have a bug when GSO is turned on which is causing iperf3 to hang. 

"runsc-xemu": { 
"path": "/usr/local/bin/runsc", 
"runtimeArgs": [ 
"--debug-log", 
"/tmp/runsc-xemu/logs/", 
"--platform=xemu", 
"--gso=false" 
] 
},