slow traafic rate

[hesam.nade...@gmail.com]

Hi there.
I have a virtualized environment with openvswitch that when performing iperf3 for link performance between hosts , I get something about 30-40Gbit/s.
I'm a newbie searching for a high speed traffic generator which can generate this traffic which be by multiple flows ( I mean it don't be stick with only a flow like iperf. I need changing IP addresses,Port Numbers,...)
Someone told me about ostinato but its performance is poor in the tests. I just get about only about 200Mbit/s and about 150000fps
I read the web about 10Gbit but in my tests its far way. It's very very far away even from iperf.
If it helps I'm running on opensuse 13.2 , kernel 3.16.7
Whats wrong. Could anybody help?

How could I reach 30 40 Gb/s?

[Carlos G Mendioroz]

What kind of load does your CPU have when doing this ?
200MB/150kfps gives ~170 bytes per frame. I would suggest using larger
frames if what you want is BW.
What frame size is iperf using in your tests ?

-Carlos

hesam.nade...@gmail.com @ 05/12/2015 10:34 -0300 dixit:

> --
> Get Ostinato News and Updates on Twitter - Follow @ostinato
> (http://twitter.com/ostinato)
> ---
> You received this message because you are subscribed to the Google
> Groups "ostinato" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to ostinato+u...@googlegroups.com
> <mailto:ostinato+u...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

--
Carlos G Mendioroz <tr...@huapi.ba.ar> LW7 EQI Argentina

[hesam.nade...@gmail.com]

In my tests cpu usage by drone proccess is about 12-13% 

I am setting the frame size to 1500 and if I be correct , 150000 x 1500 byte=225000000 B= ~ 200MB/s or 1600Mb/s

[Carlos G Mendioroz]

Oops, I wrote MB but you had stated Mbit/s :)

So now we are at 1.6 Gb/s :)

Given that this is a virtualized environment, I would try to get the CPU
usage of the hardware resource that you are using, and not the
percentage used bu drone of who knows what vision of available CPU.

Would you please describe the whole hardware and topology ?

hesam.nade...@gmail.com @ 05/12/2015 14:33 -0300 dixit:

> In my tests cpu usage by drone proccess is about 12-13%
> I am setting the frame size to 1500 and if I be correct , 150000 x 1500
> byte=225000000 B= ~ 200MB/s or 1600Mb/s
>
> On Saturday, December 5, 2015 at 8:38:37 PM UTC+3:30, Carlos G Mendioroz
> wrote:
>
> What kind of load does your CPU have when doing this ?
> 200MB/150kfps gives ~170 bytes per frame. I would suggest using larger
> frames if what you want is BW.
> What frame size is iperf using in your tests ?
>
> -Carlos
>

> hesam.nade...@gmail.com <javascript:> @ 05/12/2015 10:34 -0300 dixit:

> > an email to ostinato+u...@googlegroups.com <javascript:>
> > <mailto:ostinato+u...@googlegroups.com <javascript:>>.

> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.
>
> --
> Carlos G Mendioroz <tr...@huapi.ba.ar <javascript:>> LW7 EQI
> Argentina

[hesam.nade...@gmail.com]

I have attached pictures of stream configurations for you

Thanks :)

[hesam.nade...@gmail.com]

I have attached pictures of stream configurations for you

Thanks :)

On Saturday, December 5, 2015 at 6:21:57 PM UTC+3:30, hesam.nade...@gmail.com wrote:

[Carlos G Mendioroz]

Try to make a large (say 100) packets in the stream, see if that changes
the thing. Also, keep the discussion on the list, I'm leaving for a
while and someone else can also comment...

-Carlos

hesam.nade...@gmail.com @ 05/12/2015 15:23 -0300 dixit:

[hesam.nade...@gmail.com]

Hello again.

Today I did what you said but still no significant changes.

about 170kfps and 250MB/s 2Gb/a. It's still far from 10Gb/s

[Srivats P]

Some questions -
* How many vCPUs does your drone VM have?
* What is the CPU utilization of each vCPU?
* Are you running iperf on the same VM on which drone is running? With
no change in VM resourcing/provisioning?
* Are there any differences in hosts or topology between when you run
iperf and when you run Ostinato?
* What vNIC type are you using on drone and the receving VM?
* Instead of picture of stream configuration, save the stream and send it
* Send the full drone console log from when you start drone, including
when you start the transmission upto when you stop the transmission
* Instead of using a Ubuntu VM, can you try the vDrone appliance?

Srivats

> email to ostinato+u...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.



--

http://ostinato.org/
@ostinato

[hesam.nade...@gmail.com]

1)Actually the drone , ostantino , mininet and all are running on the same virtual machine. virtual machine is an opensuse 13.2 , kernel 3.16.7 , 12Gs of memory  and corei7 3610qm , 8 vcpu. All of drone and other are running within linux namespaces so they have access to all system resources of opensuse except that they have their own virtual network stack.

2)Cpu utilization of drone is nearly 12% shown in kde system monitor. each virtual cpu has its ups and downs but nearly two of in time are about 90-100%  and vcpus change their shares in time.

3) Yes, iperf and ostantino and drone are running within the same environment and resources and no changes between runs.

4) No, there's no change between hosts or topologies betweeen runs of iperf and ostinato.

5-6) I don't know which exact type of network is being used in mininet for virtual hosts,but i think it uses veth pairs.
I 'm taking a recording of steps and upload it for you.

7) If I use virtual appliance then I need to integrate drone in my mininet topolgy so it's somehow hard and time consuming to me.

[Carlos G Mendioroz]

I was about to ask... how many cores do you have.
The 12% seems to be 100% of one core. I guess you might be getting a top
of performance for a one threaded task.
I don't know for sure but Drone used to be single threaded...

hesam.nade...@gmail.com @ 07/12/2015 15:41 -0600 dixit:

> 1)Actually the drone , ostantino , mininet and all are running on the
> same virtual machine. virtual machine is an opensuse 13.2 , kernel
> 3.16.7 , 12Gs of memory and corei7 3610qm , 8 vcpu. All of drone and
> other are running within linux namespaces so they have access to all
> system resources of opensuse except that they have their own virtual
> network stack.
> 2)Cpu utilization of drone is nearly 12% shown in kde system monitor.
> each virtual cpu has its ups and downs but nearly two of in time are
> about 90-100% and vcpus change their shares in time.
> 3) Yes, iperf and ostantino and drone are running within the same
> environment and resources and no changes between runs.
> 4) No, there's no change between hosts or topologies betweeen runs of
> iperf and ostinato.
> 5-6) I don't know which exact type of network is being used in mininet
> for virtual hosts,but i think it uses veth pairs.
> I 'm taking a recording of steps and upload it for you.
> 7) If I use virtual appliance then I need to integrate drone in my
> mininet topolgy so it's somehow hard and time consuming to me.
>
>

> <https://groups.google.com/d/optout>.

> >>
> >> --
> >> Carlos G Mendioroz <tr...@huapi.ba.ar> LW7 EQI Argentina
> >
> > --
> > Get Ostinato News and Updates on Twitter - Follow @ostinato
> > (http://twitter.com/ostinato)
> > ---
> > You received this message because you are subscribed to the
> Google Groups
> > "ostinato" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send an

> > email to ostinato+u...@googlegroups.com <javascript:>.

> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
>
> --

> http://ostinato.org/
> @ostinato

[hesam.nade...@gmail.com]

None of cpus at 100%
I have attached the recording. link at : http://filepi.com/i/2jguSBM

[Carlos G Mendioroz]

It does not matter which CPU (physical) the (logical) vcpu is mapped to,
if a task is mono-threaded, it can not go faster than one proc in your
physical one. 1/8 = 12.5%


hesam.nade...@gmail.com @ 07/12/2015 16:46 -0600 dixit:

> None of cpus at 100%
> I have attached the recording. link at : http://filepi.com/i/2jguSBM
>
> On Tuesday, December 8, 2015 at 1:18:56 AM UTC+3:30, Carlos G Mendioroz
> wrote:
>
> I was about to ask... how many cores do you have.
> The 12% seems to be 100% of one core. I guess you might be getting a
> top
> of performance for a one threaded task.
> I don't know for sure but Drone used to be single threaded...
>

> hesam.nade...@gmail.com <javascript:> @ 07/12/2015 15:41 -0600 dixit:

> > an email to ostinato+u...@googlegroups.com <javascript:>
> > <mailto:ostinato+u...@googlegroups.com <javascript:>>.

> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --

> Carlos G Mendioroz <tr...@huapi.ba.ar <javascript:>> LW7 EQI
> Argentina
>

[hesam.nade...@gmail.com]

If we accept that it comes from cpu speed then how to reach milionsfps per second?
in documentation it's said ostinato is able to produce 10Gs/s and milion fps/s.

3610qm cpu clock is 3.2Ghz and benchmark score is ~8000.
top market cpus clock are ~3.9Ghz and benchmark score ~14000

With this numbers it seems not being able to produce Mfps.
Something is oddly wrong.
Is there any solution to overcome this mono thread? Any esay way to make it multi-threaded?

[Srivats P]

Which documentation are you referring to which claims 10Gbps and million fps?

Currently there's no way to make it multithreaded - unless you want to
try changing code.

One thing you could try is use jumbo frames assuming your veth
interfaces do - Ostinato supports frame size upto 16K.

The iperf speeds you claim are they client reported speed or server
reported speed?

Srivats

[hesam.nade...@gmail.com]

http://www.slideshare.net/mobile/plvision/10-g-packet-processing-with-dpdk

http://www.ntop.org/pf_ring/building-a-10-gbit-traffic-generator-using-pf_ring-and-ostinato/

https://groups.google.com/forum/m/#!topic/ostinato/ZdiR2ZXdt0c

Iperf stats are reported by both of client and server. You may want to have a look at my recording upload link at the previous post.

[Srivats P]

All the links you listed are extensions done to Ostinato by other
people which is not present in the official code - you need to follow
up with the people behind those links on how to get their changes if
you want.
I just saw the video you posted and I see what you are saying. While
I'm very tempted to investigate how iperf is giving better performance
than Ostinato (TSO and less context switching between user and kernel
are possible candidates, also the fact that iperf is multithreaded),
the fact of the matter is I just don't have the time for it.

If some one on this list wants to undertake this investigation, I can
chip in and discuss, but I don't have the time to investigate this
myself. Also there is the matter that I don't really have access to a
multi-core server with that kind of specs.

One thing I would request is if you could let me know how much is the
iperf performance with a single thread - it will give me a rough idea
on how much impact multi-threading has.

Srivats

[Carlos G Mendioroz]

Hi Srivats,
I was looking and it seems that even though Iperf has multithreading
performance, it would seem that by default it does not use it, or it
only uses 2, one reader and one writter.

I'm kind of positive that CPU is the limiting factor here, but as I
privatelly told Hesam (ok to address you this way ?), I'm out of my lab
this week.

-Carlos

Srivats P @ 10/12/2015 08:45 -0600 dixit:

[Srivats P]

Carlos,

12 * 8 = 96 seemed like a giveaway, but if you watch Hesam's video,
you'll see when Drone is running, one of the core's CPU utilization is
around 37% while the rest are insignificant. This would indicate that
drone is not able to utilize even one core completely - indicating
bottleneck is not CPU I guess. When iperf is running, one of the core
goes up to 15% while the rest are insignificantly loaded - so you are
right iperf in this instance at least is not running multithreaded.

I'm not sure how the per core utilization % is aggregated into a
single aggregate per-process cpu utilization % though.

Srivats

--
http://ostinato.org/
@ostinato

[Carlos G Mendioroz]

The problem with VMs is that the vcpu to pcpu mapping is not fixed, so
even a 100% core in one VM may not reflect as a 100% (or even 50%) core
usage in the host.
Given that in this case there is a two layer stack (physical running
ubuntu VM running lab as namespaces) there might be even more issues,
though I don't think so.
But the 12-13% cpu usage reported by drone in a 8 way CPU sounds a lot
as CPU bounding to me :)

-Carlos

Srivats P @ 10/12/2015 10:09 -0600 dixit:

[Srivats P]

I admit I'm not very conversant about how vCPUs are mapped to pCPUs
but whatever v/p CPU limitation drone has in the environment, iperf
should too - so why such a difference between them?

Srivats

--
http://ostinato.org/
@ostinato

[Carlos G Mendioroz]

Beats me.
iperf has to have a more efficient way of generating traffic.
That's what I would infer from the experiment...

-Carlos

Srivats P @ 11/12/2015 09:57 -0600 dixit: