netsniff-ng dropping packets at 80Mbps

[Unnikannan Nair, Jishnu]

Hi

I’m trying to build a network sniffer for UDP multicast streams using netsniff-ng. I have deployed two linux systems (Ubuntu server 14.04 LTS) on an ESXi one has the sniffer and other has tcpreplay sending packets at needed Mbps. The configuration is:

Sniffer:

·         Intel i7 3770 3.4GHz

·         2 cores added to the sniffer VM

·         16GB RAM

 

The sniffer job is  as follows :

Sudo netsniff-ng –I eth1 –out=/share/job1/ --prefix=”job1_”  --interval 2MiB –ring-size 3GiB –prio-high –f “dst net 224.10.10” –user 1000 –group 1000 –b 1

 

The tcp replay job is as follows

tcpreplay –i eth1 –M 80Mbps –loop 100 –preload-pcap –sleep-accel=1  /home/caps/*.pcap

 

Tcpreplay sends 10 pcap files , 100 times i.e. total packets send is 10487400 @ 80.77Mbps but netsniff-ng received only 504813

 

TCPREPLAY:

Actual: 10487400 packets (5229757800 bytes) sent in 481.67 seconds.            Rated: 10857554.0 bps, 82.84 Mbps, 21773.00 pps

Statistics for network device: eth1

        Attempted packets:         10487400

        Successful packets:        10487400

        Failed packets:            0

        Retried packets (ENOBUFS): 0

        Retried packets (EAGAIN):  0

 

NETSNIFF_NG:

504813  packets incoming (9982587 unread on exit)

5929469  packets passed filter

4557931  packets failed filter (out of space)

43.4610% packet droprate

559  sec, 715362 usec in total

Cannot set NIC flags!

 

I have increased the kernel RX Cache to 1GB in /etc/sysctl.conf, but it did not have any effect. Could someone give me some information as to how to fix this?? Also the last line “Cannot set NIC Flag” what does that mean?? Is that an error?? My target is to run two instance of netsniff on eth1 and eth2 with different filters and bind them on CPU 1 and CPU 2 later on.

 

Regards

Jishnu

 

 

Jishnu Unnikannan Nair | Systems Engineer

NOV Rig Systems

Systems & Controls | Drilling data Center

Lagerveien 8| 4033 Stavanger, Norway

T   +475.181.8181

M +473.819.4208

E   jishnu.unni...@nov.com

nov.com

Connect with us on Facebook | LinkedIn | Twitter

The information contained in this transmission is for the personal and confidential use of the individual or entity to which it is addressed. If the reader is not the intended recipient, you are hereby notified that any review, dissemination, or copying of this communication is strictly prohibited. If you have received this transmission in error, please notify the sender immediately.

 

 

[Daniel Borkmann]

On 07/20/2015 01:45 PM, Unnikannan Nair, Jishnu wrote:
> Hi
> I'm trying to build a network sniffer for UDP multicast streams using netsniff-ng. I have deployed two linux systems (Ubuntu server 14.04 LTS) on an ESXi one has the sniffer and other has tcpreplay sending packets at needed Mbps. The configuration is:
> Sniffer:
>

> * Intel i7 3770 3.4GHz
>
> * 2 cores added to the sniffer VM
>
> * 16GB RAM

>
>
>
> The sniffer job is as follows :
>

> Sudo netsniff-ng -I eth1 -out=/share/job1/ --prefix="job1_" --interval 2MiB -ring-size 3GiB -prio-high -f "dst net 224.10.10" -user 1000 -group 1000 -b 1

Hmm, what version do you use? Have you tried a more recent version?

Also, I fail to see that you're invoking it in 'silent' mode (-s).

>
>
> The tcp replay job is as follows
>

> tcpreplay -i eth1 -M 80Mbps -loop 100 -preload-pcap -sleep-accel=1 /home/caps/*.pcap

> nov.com<http://www.nov.com/>
> Connect with us on Facebook<https://www.facebook.com/NationalOilwellVarco> | LinkedIn<http://www.linkedin.com/company/national-oilwell-varco?trk=tyah&trkInfo=tarId%3A1405455811306%2Ctas%3Anational+oilwell%2Cidx%3A2-1-4> | Twitter<https://twitter.com/NOVGlobal>

[Unnikannan Nair, Jishnu]

Hi
I'm running the latest version 0.5.9, and im not running it in silent mode as of now just to see the dropped packets information when the capture is complete. I intend to run it as silent when all issues are solved :)

Jishnu Unnikannan Nair | Systems Engineer

[Daniel Borkmann]

On 07/20/2015 02:06 PM, Unnikannan Nair, Jishnu wrote:
> Hi

> I'm running the latest version 0.5.9, and im not running it in silent mode as of now just to see the dropped packets information when the capture is complete. I intend to run it as silent when all issues are solved :)

Well, dumping everything through the dissector code and expecting it
to serve the packet as fast as possible and clear up the ring slot
again certainly seems a false expectation. Besides, even if you terminate
in silent mode, it's giving you stats:

netsniff-ng -i any -s
Running! Hang up with ^C!

35 packets incoming (0 unread on exit)
35 packets passed filter
0 packets failed filter (out of space)
0.0000% packet droprate
6 sec, 178111 usec in total
...

Cheers,
Daniel

[Unnikannan Nair, Jishnu]

So are you saying that it's impossible to dump pcap files at this rate??

regards

Jishnu Unnikannan Nair | Systems Engineer
NOV Rig Systems
Systems & Controls | Drilling data Center
Lagerveien 8| 4033 Stavanger, Norway
T +475.181.8181
M +473.819.4208
E jishnu.unni...@nov.com
nov.com

Connect with us on Facebook | LinkedIn | Twitter

The information contained in this transmission is for the personal and confidential use of the individual or entity to which it is addressed. If the reader is not the intended recipient, you are hereby notified that any review, dissemination, or copying of this communication is strictly prohibited. If you have received this transmission in error, please notify the sender immediately.

-----Original Message-----
From: Daniel Borkmann [mailto:bork...@iogearbox.net]
Sent: 20. juli 2015 02:13 PM
To: Unnikannan Nair, Jishnu
Cc: netsn...@googlegroups.com
Subject: Re: [netsniff-ng] netsniff-ng dropping packets at 80Mbps

[Daniel Borkmann]

On 07/20/2015 02:21 PM, Unnikannan Nair, Jishnu wrote:
> So are you saying that it's impossible to dump pcap files at this rate??

Please, that's not what I wrote in my email.

I wrote that you should use -s.

[Unnikannan Nair, Jishnu]

Hi
Its a bit confusing for me now. With -s option it gives me the following

Running! Hang up with ^C!

^C

Cannot set NIC flags!


It doesn’t give any statistics. Is this something wrong with the NICs?

regards


Jishnu Unnikannan Nair

-----Original Message-----
From: Daniel Borkmann [mailto:bork...@iogearbox.net]
Sent: 20. juli 2015 02:26 PM
To: Unnikannan Nair, Jishnu
Cc: netsn...@googlegroups.com
Subject: Re: [netsniff-ng] netsniff-ng dropping packets at 80Mbps

[Unnikannan Nair, Jishnu]

Hi
I have double checked, it seems if I run with "-s" option with the application saving files on to a folder "--out /share/capture/job/" will not give any statistics up on stopping it using ^C.
And it is giving a "Cannot set NIC Flags" message at the end. So I'm not able to check if the sniffer has captured all the packets. Could you please help me with this.

-----Original Message-----
From: Daniel Borkmann [mailto:bork...@iogearbox.net]
Sent: 20. juli 2015 02:26 PM
To: Unnikannan Nair, Jishnu
Cc: netsn...@googlegroups.com
Subject: Re: [netsniff-ng] netsniff-ng dropping packets at 80Mbps

[Daniel Borkmann]

On 07/20/2015 04:58 PM, Unnikannan Nair, Jishnu wrote:
> Hi
> I have double checked, it seems if I run with "-s" option with the application saving files on to a folder "--out /share/capture/job/" will not give any statistics up on stopping it using ^C.
> And it is giving a "Cannot set NIC Flags" message at the end. So I'm not able to check if the sniffer has captured all the packets. Could you please help me with this.

Could you pull/recompile the latest repo? We don't dump the actual
error, which we should do to easier identify issues. If you cannot
recompile, the alternative is to strace into it, so we can see what
error you get that lets netsniff-ng bail out?

Thanks,
Daniel

[Unnikannan Nair, Jishnu]

Hi
I have re-complied the source form the GIT REPO and installed netsniff again. I got the same message when I close the application "Cannot set NIC flags (operation not permitted)".
So I ran strace on a running instance of netsniff-ng and I have attached it. Hope you can find the reason for the error.
Regards
Jishnu

-----Original Message-----
From: Daniel Borkmann [mailto:bork...@iogearbox.net]
Sent: 20. juli 2015 06:54 PM
To: Unnikannan Nair, Jishnu
Cc: netsn...@googlegroups.com
Subject: Re: [netsniff-ng] netsniff-ng dropping packets at 80Mbps

[Vadim Kochan]

On Tue, Jul 21, 2015 at 07:56:45AM +0000, Unnikannan Nair, Jishnu wrote:
> Hi

> I have re-complied the source form the GIT REPO and installed netsniff again. I got the same message when I close the application "Cannot set NIC flags (operation not permitted)".
> So I ran strace on a running instance of netsniff-ng and I have attached it. Hope you can find the reason for the error.
> Regards
> Jishnu
>
>
>

Hi,

I assume you do not have enough permissions probably because of -u 1000 -g 1000 ?

Did you try use only super user ?

Also as I understand in silent mode you should get stats after you
terminate sniffing by Ctr-C, so would you provide these values & check
if there is some drop rate ? Also may be you can try sniff w/o pcap
filter and check interface stats via ifpps or some other tool which you like ?

Regards,
Vadim Kochan

[Unnikannan Nair, Jishnu]

Hi

When I removed -u 1000 -g 1000 , "Cannot set NIC flags " is not displayed anymore,but no statistics are shown when the sniffer is stopped. I have run the application as root using "sudo su -" but it still doesn't show the statistics.
About using iffps I think it shows how many packets are received by the kernel, I wanted to know how many packets are properly dumped by the sniffer into the pcap file (hoping that that's the inforamtion netsniff-ng gives).

Regards
Jishnu


-----Original Message-----
From: Vadim Kochan [mailto:vad...@gmail.com]
Sent: 21. juli 2015 11:17 AM
To: Unnikannan Nair, Jishnu
Cc: 'Daniel Borkmann'; netsn...@googlegroups.com; vad...@gmail.com
Subject: Re: [netsniff-ng] netsniff-ng dropping packets at 80Mbps

[Vadim Kochan]

On Mon, Jul 20, 2015 at 02:58:21PM +0000, Unnikannan Nair, Jishnu wrote:
> Hi
> I have double checked, it seems if I run with "-s" option with the application saving files on to a folder "--out /share/capture/job/" will not give any statistics up on stopping it using ^C.
> And it is giving a "Cannot set NIC Flags" message at the end. So I'm not able to check if the sniffer has captured all the packets. Could you please help me with this.

Daniel,

I might be wrong but I looked into the code and see this condition in
the netsniff-ng.c:

1097 if (!(ctx->dump_dir && ctx->print_mode == PRINT_NONE)) {
1098 sock_rx_net_stats(sock, frame_count);

which seems that really netsniff-ng will do not print stats in silent
mode and if dump to the folder ?

Regards,
Vadim

[Vadim Kochan]

On Tue, Jul 21, 2015 at 11:00:10AM +0000, Unnikannan Nair, Jishnu wrote:
>
>
> Hi
>

> When I removed -u 1000 -g 1000 , "Cannot set NIC flags " is not displayed anymore,but no statistics are shown when the sniffer is stopped. I have run the application as root using "sudo su -" but it still doesn't show the statistics.
> About using iffps I think it shows how many packets are received by the kernel, I wanted to know how many packets are properly dumped by the sniffer into the pcap file (hoping that that's the inforamtion netsniff-ng gives).
>
>
> Regards
> Jishnu
>
>

At least you can try to dump each pcap file and netsniff-ng
should print the amount of packets from pcap file ?

[Daniel Borkmann]

Hmm, fair enough. Maybe this should be reworked. We dump intermediate stats
in print_pcap_file_stats(), which you should see with -V option in this mode.
I.e. it shows [captured pkts/drops] after each rotation. Afaik, whenever you
fetch PACKET_STATISTICS via getsockopt(2), they're cleared in the kernel again.

[Unnikannan Nair, Jishnu]

Hi
I didn't get what you meant, could you please explain a bit.

Regards
Jishnu

-----Original Message-----
From: Vadim Kochan [mailto:vad...@gmail.com]
Sent: 21. juli 2015 01:21 PM
To: Unnikannan Nair, Jishnu
Cc: 'Vadim Kochan'; 'Daniel Borkmann'; netsn...@googlegroups.com
Subject: Re: [netsniff-ng] netsniff-ng dropping packets at 80Mbps