However, in any vm sending traffic through this netvm I can only get around 2Gbit max using the same iperf command against the same server.

I have this problem both with appvms using the *same* Fedora template as the netvm, and also in Windows HVM.

Anybody have experience with Qubes in a 10Gbe network?

Interested to find out too. Have you tried from FirewallVM?

Could you also test what happen when during the load test you start a disposable VM? Does it drop (I am using a Qubes server as a firewall, and my kids complain it freeze their game for few seconds when I start DispVM (my intranet card is in an appVM with the PCI NIC card attached).

On Wed, 31 Jan 2018, Jarle Thorsen wrote:

> onsdag 31. januar 2018 11.12.33 UTC+1 skrev Ilpo Järvinen følgende:
> > On Wed, 31 Jan 2018, Jarle Thorsen wrote:
> > > onsdag 31. januar 2018 10.50.09 UTC+1 skrev Jarle Thorsen følgende:
> > > > My netvm (Fedora 26 template) has a 10gbe network card, and from
> > > > within the netwm I have no problem saturating the 10Gbit link using
> > > > iperf to an external server.
> > > > However, in any vm sending traffic through this netvm I can only get
> > > > around 2Gbit max using the same iperf command against the same server.
> > > >
> > > > I have this problem both with appvms using the *same* Fedora template
> > > > as the netvm, and also in Windows HVM.
> > > >
> > > > Anybody have experience with Qubes in a 10Gbe network?
> > >
> > > I do notice the the process ksoftirqd in the netvm is running at 100%
> > > cpu during the iperf test from the appvm. I'm guessing this is related
> > > to my problem?
> >
> > SG being disabled for the qubes inter-vm interfaces (due to an unknown
> > reason as Marek didn't anymore remember the details for that change) might
> > have some(/huge?) impact on performance as it prevents using some
> > high-speed features of the kernel.
>
> Not quite sure what "SG" is in this context. Is there any settings I can
> experiment with?

> Are you talking about enabling sg on the virtual network device in the
> netvm?
>
> Something like "sudo ethtool -K vif12.0 sg on" ?

> Interested to find out too. Have you tried from FirewallVM?

Same slow performance when iperf is run from a FirewallVM that is connected to the netvm.

> Could you also test what happen when during the load test you start a disposable VM? Does it drop

Running iperf in the netvm, and starting a disposable vm using the same netvm there is no performance drop. (I get 8+ Gbit running one thread, and about 9,5Mbit when using multiple threads)

> Scatter-Gather.
>
> > Are you talking about enabling sg on the virtual network device in the
> > netvm?
> >
> > Something like "sudo ethtool -K vif12.0 sg on" ?
>
> Yes. For both that and the eth0 in appvm.

On Wed, 31 Jan 2018, Jarle Thorsen wrote:

> Ilpo Järvinen:
> > Scatter-Gather.
> >
> > > Are you talking about enabling sg on the virtual network device in the
> > > netvm?
> > >
> > > Something like "sudo ethtool -K vif12.0 sg on" ?
> >
> > Yes. For both that and the eth0 in appvm.
>
> This made a huge performance boost! (single threaded iperf went from
> 1.34 Gbit/sec to 3.75 Gbit/sec in appvm after enabling SG)

> I'm still far away from the speed I get inside the netvm though... Maybe
> the netvm needs more than 2 vcpus to handle the stress from appvms?

> Please also check that GSO (generic-segmentation-offload) is on at
> the sending appvm eth0 (I don't remember if the depency logic causes it to
> get toggled off when SG was off'ed and cannot check it ATM myself).

> > I'm still far away from the speed I get inside the netvm though... Maybe
> > the netvm needs more than 2 vcpus to handle the stress from appvms?
>
> Could be.

> I'm not sure how your appvm is connected to netvm if there's firewall vm
> in between that would also be relevant.

 
> I guess you can try e.g. vmstat 1 in all vms through which the traffic
> passes through to see if any of them is saturating CPU.

Thanks for testing and for the SG module I didn't know before this thread. I'll have to dig into my pb...

> It sounds a bit ambitious to run 10gb per sec from one VM through
> another and onto the wire.  I suspect you are memory speed limited
> if you are using a straightforward desktop pc.

I'm not sure what is the limiting factor (memory speed, xen overhead?), but I just did an iperf test between netvm and appvm only, and I still maxed out at 4 Gbit/s. This test takes the network card out of the equation and only tests the speed between the vms. Maybe somebody can do similar tests on their system?

> Do you know if anyone has achieved this?

I don't know.

On Wed, 31 Jan 2018, Jarle Thorsen wrote:

> I'd next try to tweak the txqueuelen (at the netvm side):
>   sudo ifconfig vifxx.0 txqueuelen xxxx
>
> Appvm side (eth0) seems to have 1000 but the other side (vifxx.0) has
> only 64 by default that seems a bit small for high-performance transfers.

> I found this:
> https://wiki.xenproject.org/wiki/Xen-netback_and_xen-netfront_multi-queue_performance_testing

> It might be that roughly 4Gbps might be what you can get for cross-vm with
> one flow (but those results are quite old).
>
> I guess that there are by default two queues per vif (based on the kernel
> thread naming) which would explain why going beyond 2 VCPUs didn't help
> any. ...Now we only just need to figure out how to configure the number of
> queues to see if that has some impact on performance (ethtool seems to be
> unable to do that).
>
> ...And of course you'll need to use more than one flow to utilize all
> those queues anyway but I guess you tested also the inter-vm test with
> more than one flow?

> Can you try if you get better throughput between a proxy vm and an appvm
> using this kind of topology?
>
> sys-net <-> iperf-srv (proxyvm) <-> iperf-cli (appvm)
>
> I could push ~10Gbps with one flow and slightly more with more parallel
> flows between them.

On Fri, 2 Feb 2018, Jarle Thorsen wrote:

> Ilpo Järvinen:
> > Can you try if you get better throughput between a proxy vm and an appvm
> > using this kind of topology?
> >
> > sys-net <-> iperf-srv (proxyvm) <-> iperf-cli (appvm)
> >
> > I could push ~10Gbps with one flow and slightly more with more parallel
> > flows between them.
>
> Great find Ilpo! Did you have to do some iptables-trickery for this
> testing? I have ping working between proxy and appvm, but iperf and nc
> both tell me no route to host?

> > Great find Ilpo! Did you have to do some iptables-trickery for this
> > testing? I have ping working between proxy and appvm, but iperf and nc
> > both tell me no route to host?
>
> Yes, I did (it replies with ICMP by default). You'll need to fill in the
> vif IP-address to this command:
>
> sudo iptables -I INPUT 1 -i vif+ -p tcp -d ***proxyvmIPhere*** -j ACCEPT