AF_PACKET vs. PF_RING

[Security Reseacher]

To whom it may concern,

Recently, I noticed in the change logs that Zeek/Bro and Suricata are going to capture traffic using AF_PACKET instead of PF_RING. Having said that, I have a few questions:

1. What was the motivation for switching over to AF_PACKET?
2. Did you perform any performance comparisons between AF_PACKET and PF_RING? (Packet drop rate, speed, etc.). If so, can they be shared?

Thank you for your time and help.

[Wes Lambert]

Hi Security Reseacher,

We don't have any metrics to share, however, we have noticed a decrease in overall packet loss and improved performance using afpacket vs pfring.

This, in turn, would mean that you can sustain successful capture with minimal loss at speeds higher than attainable with pfring with the same or similar resources.

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/52442a3d-55e0-413e-a0ad-9ed825de860e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

https://twitter.com/therealwlambert

https://securityonion.net/

[Security Reseacher]

Hi Wes,

Thank you for sharing that information. I have a follow-up question; I understand that Security Onion never had PF_RING Zero Copy, however, did you by any chance compare AF_PACKET's performance to PF_RING Zero Copy?

Again, thank you for your time.

[Wes Lambert]

I have not.

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/9e062dfe-157e-4312-a2ac-43a71df7e55d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Simone Bonetti]

I tried to use af_packet, I immediately switched back to pf_ring.
pf_ring (no zero copy or anything else) has a much higher performance than af_packet.
The same machine with zeek and pf_ring I have no losses. With af_packet I have 10% losses.
pf_ring also has other wonderful features (eg pf_ring ft or ebpf) that makes no sense to give up.

[Wes Lambert]

Hi Simone,

I'd be interested to hear your testing strategy.  How have you compared the two?  We certainly want to make sure folks are receiving the best performance possible, so any feedback you have would be great!

Thanks!

Wes 

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/2aeecc3d-d738-411f-8e45-efe73f7beedc%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Simone Bonetti]

> To unsubscribe from this group and stop receiving emails from it, send an email to securit...@googlegroups.com.

>
> To post to this group, send email to securit...@googlegroups.com.
>
> Visit this group at https://groups.google.com/group/security-onion.
>
> To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/2aeecc3d-d738-411f-8e45-efe73f7beedc%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
>
>
> https://twitter.com/therealwlambert
>
> https://securityonion.net/

Hi Wes,
it's not a valid strategy. I searched some docs about af_packet configuration. It's not to easy find or understand them. Then I changed node.cfg and restarted zeek.

After 5 minutes I had 10% of drops.

Then I edited again node.cfg to the original configuration, restarted zeek and magically I had 0% drops. I had always 0% drop.

It's not the right way to measure that but it's ok for me 10% drops is a lot.

I think it's premature to switch to af_packet.
I think pf_ring offer many feature like pf_ring ft very interesting for detection. Parse a netflix flow is a nonsense. Pf_ring ft can avoid that. pf_ring zc and so on are other wonderful idea.

The issue is the pf_ring version in securityonion is very old (6.6.0). The pf_ring stable version is 7.4 (I'm using dev version 7.5). The old version in security onion cam't permit to use ntopng another wonderful tool for network visibility from the same developer team.

As I said it's not the right way to measure that but it's right for me. If you can share a valid configuration to use af_packet I'll be happy to try it.

I used this:
----begin node.cfg
[manager]
type=manager
host=localhost

[proxy0]
type=proxy
host=localhost

[proxy1]
type=proxy
host=localhost

[proxy2]
type=proxy
host=localhost

[proxy3]
type=proxy
host=localhost

[proxy4]
type=proxy
host=localhost

# https://github.com/J-Gras/bro-af_packet-plugin
[xxx-xxx6]
type=worker
host=localhost
interface=af_packet::xxx6
lb_method=custom
lb_procs=22
pin_cpus=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21
# Optional parameters for per node configuration:
af_packet_fanout_id=23
af_packet_fanout_mode=AF_Packet::FANOUT_HASH
af_packet_buffer_size=128*1024*1024
----end node.cfg

Thanks

Simone

[Doug Burks]

Hi Simone,

When you tested AF_PACKET, what version of Bro were you using?  And what version of Security Onion?

If you haven't already, you might want to try the latest version of Security Onion:

https://blog.securityonion.net/2019/05/security-onion-160461-now-available.html

When you run Setup and choose Production Mode, it should configure Bro for AF_PACKET automatically and write out the a basic node.cfg AF_PACKET config for you.

Depending on your hardware specs and traffic, you may need to tune using:

https://securityonion.readthedocs.io/en/latest/af-packet.html

https://securityonion.readthedocs.io/en/latest/performance.html

Hope that helps!

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.

To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/a263ea6c-22ea-4e0e-989d-c831c69b3634%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Doug Burks
CEO
Security Onion Solutions, LLC

[Doug Burks]

Hi Simone,

Replies inline.

On Thu, Jun 20, 2019 at 4:53 AM Simone Bonetti <pascal....@gmail.com> wrote:

Hi Doug,
I run soup command every day, so when I tried ZEEK was the last version, but I don't remember that. Probably it was 2.6.0 or 2.6.1, maybe 2.6.1.
Now I'm using zeek 2.6.2 but my zeek configuration use pf_ring.

broctl diag report:
Bro plugins:
Bro::AF_Packet - Packet acquisition via AF_Packet (dynamic, version 1.3)

# cat /opt/bro/etc/node.cfg
[manager]
type=manager
host=localhost

[proxy]
type=proxy

host=localhost

[proxy0]
type=proxy
host=localhost

[proxy1]
type=proxy
host=localhost

[proxy2]
type=proxy
host=localhost

[proxy3]
type=proxy
host=localhost

[xxxxxx-yyyyyyyyy]   
type=worker
host=localhost
interface=yyyyyyyyy
lb_method=pf_ring
lb_procs=30
pin_cpus=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29

I don't want use setup because it's in production. Tell me how to change my configuration and I'll try for you.

How much traffic are you monitoring?

What kind of hardware do you have?

How many CPUs?  How many cores?  Do you have hyperthreading enabled?

 

I ask you to revisit your choice about Security Onion and pf_ring.

The next major version of Security Onion (Hybrid Hunter) will only use AF_PACKET since it is already built into the Linux kernel (and has been for some time) and can scale to high traffic levels:

https://github.com/pevma/SEPTun
https://github.com/pevma/SEPTun-Mark-II

Thanks

Simone

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/1210eda3-4d0b-4323-8035-13b01cfe8c18%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Simone Bonetti]

Il giorno venerdì 14 giugno 2019 19:54:39 UTC+2, Doug Burks ha scritto:
> Hi Simone,
>
>
> When you tested AF_PACKET, what version of Bro were you using?  And what version of Security Onion?
>
>
> If you haven't already, you might want to try the latest version of Security Onion:
> https://blog.securityonion.net/2019/05/security-onion-160461-now-available.html
>
>
>
> When you run Setup and choose Production Mode, it should configure Bro for AF_PACKET automatically and write out the a basic node.cfg AF_PACKET config for you.
>
>
> Depending on your hardware specs and traffic, you may need to tune using:
> https://securityonion.readthedocs.io/en/latest/af-packet.html
> https://securityonion.readthedocs.io/en/latest/performance.html
>
>
> Hope that helps!
>

> --
>
> Doug Burks
> CEO
> Security Onion Solutions, LLC

Hi Doug,
my security onion nodes are always up to date :-).
I tried af_packet with the version of zeek 2.6.0 or 2.6.1.
As you suggested I tried again with version 2.6.2, there is no story.
pf_ring has 0 losses, while af_packet has losses that vary between 1.5% and 4.5%. In this period our traffic is reduced by half that's why af_packet reports lower losses than the previous time.

I hope you think about eliminating pf_ring from security onion. Below are two interesting links on two innovations of the pf_ring family:
https://www.ntop.org/products/packet-capture/pf_ring/pf_ring-ft-flow-table/
https://www.ntop.org/announce/introducing-libebpfflow-packet-less-network-traffic-and-container-visibility-based-on-ebpf/

Thanks

Simone

[Doug Burks]

Hi Simone,

How much traffic are you monitoring?

What kind of hardware do you have?

How many CPUs?  How many cores?  Do you have hyperthreading enabled?
 

I'm fairly confident that AF_PACKET should be able to handle your traffic load as Michał Purzyński at Mozilla has been running AF_PACKET at 20Gbps and above for a while now:
https://github.com/pevma/SEPTun
https://github.com/pevma/SEPTun-Mark-II

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/f8ec1e6b-c219-4e85-b523-9946dcfc07d6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Simone Bonetti]

Hi Doug

> How much traffic are you monitoring?

broctl capstats
Interface kpps mbps (10s average)
----------------------------------------
localhost/ens6 333.7 1783.2
This is low traffic for me, usually we double that

> What kind of hardware do you have?

HP DL 585 G6 very old HW :-) RAM: 128G
nic: Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

> How many CPUs?  How many cores?  Do you have hyperthreading enabled?

4, 24, yes

AF_Packet don't scale for me :-)

Simone

[Doug Burks]

Hi Simone,

It sounds like you've already made up your mind and aren't willing to try any tuning suggestions for AF_PACKET.  If that's the case, OK.  

However, if you'd be willing to experiment with AF_PACKET with an open mind, we might be able to offer some tuning suggestions.  Please let us know if we can help.  

Thanks! 

[Simone Bonetti]

Il giorno giovedì 20 giugno 2019 16:56:08 UTC+2, Doug Burks ha scritto:

> On Thu, Jun 20, 2019 at 9:24 AM Simone Bonetti <pascal...@gmail.com> wrote:
> Hi Simone,
>
>
> It sounds like you've already made up your mind and aren't willing to try any tuning suggestions for AF_PACKET.  If that's the case, OK.  
>
>
> However, if you'd be willing to experiment with AF_PACKET with an open mind, we might be able to offer some tuning suggestions.  Please let us know if we can help.  
>
>
> Thanks!

Hi Doug you can't say that.
I tried as you suggest me and I had many drops with af_packet but none with pf_ring.
This is a fact.
If you have some suggestions I'll be happy to try again.

Thanks

Simone

[Doug Burks]

Hi Simone,

Thanks for being open to further experimentation with AF_PACKET!  :)

I would suggest starting with a simple Bro configuration without pin_cpus just to get a performance baseline.  That would look something like this:

[logger]
type=logger
host=localhost

[manager]
type=manager
host=localhost

[proxy]
type=proxy
host=localhost

[HOSTNAME-INTERFACE]  
type=worker
host=localhost
interface=INTERFACE
lb_method=custom
lb_procs=18
af_packet_fanout_id=31
af_packet_fanout_mode=AF_Packet::FANOUT_HASH
af_packet_buffer_size=128*1024*1024

Also note that config includes a [logger] stanza which you may not have previously had.

Once you have a performance baseline with a config like the above, then I would suggest experimenting with different settings for lb_procs, pin_cpus, and af_packet_buffer_size.  For pin_cpus, you may want to try pinning to only real physical cores and avoid hyperthreaded cores to see if that makes a difference.

You may also want to look at some of the tuning suggestions that enabled Michał Purzyński at Mozilla to reach 20Gbps:

https://github.com/pevma/SEPTun
https://github.com/pevma/SEPTun-Mark-II

Hope that helps!

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/c0fcff24-1074-4c3b-9448-46f38fa5d265%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Simone Bonetti]

Here we are.
Hi Doug, I open my mind I hope you do the same :)

My HW is an old HP proliant DL585 G6, CPU 4 x Six-Core AMD Opteron(tm) Processor 8439 SE, RAM 128G
nic: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection
traffic link: usually 2-4Gbs even more

I tried first my configuration with af_packet then pf_ring
I use more proxies as suggested in this paper:
http://commons.lbl.gov/download/attachments/120063098/100GIntrusionDetection.pdf

(af_packet)
# cat node.cfg
[manager]
type=manager
host=localhost

[logger]
type=logger

host=localhost

[proxy0]
type=proxy
host=localhost

[proxy1]
type=proxy
host=localhost

[proxy2]
type=proxy
host=localhost

[proxy3]
type=proxy
host=localhost

[proxy4]
type=proxy
host=localhost

[pippo-ens6]
type=worker
host=localhost
interface=ens6
lb_method=custom
lb_procs=22
pin_cpus=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21
af_packet_fanout_id=31
af_packet_fanout_mode=AF_Packet::FANOUT_HASH
af_packet_buffer_size=128*1024*1024


Now the results
root@pippo:/opt/bro/etc# broctl top; date
Name Type Host Pid VSize Rss Cpu Cmd
logger logger localhost 11523 504M 116M 17% bro
manager manager localhost 11609 205M 103M 23% bro
proxy0 proxy localhost 11771 198M 96M 5% bro
proxy1 proxy localhost 11772 199M 97M 5% bro
proxy2 proxy localhost 11776 195M 92M 0% bro
proxy3 proxy localhost 11780 197M 95M 5% bro
proxy4 proxy localhost 11781 197M 94M 0% bro
pippo-ens6-1 worker localhost 12117 379M 276M 41% bro
pippo-ens6-2 worker localhost 12092 390M 279M 29% bro
pippo-ens6-3 worker localhost 12144 387M 277M 35% bro
pippo-ens6-4 worker localhost 12101 377M 275M 23% bro
pippo-ens6-5 worker localhost 12128 376M 274M 17% bro
pippo-ens6-6 worker localhost 12140 391M 281M 88% bro
pippo-ens6-7 worker localhost 12133 383M 273M 29% bro
pippo-ens6-8 worker localhost 12200 375M 273M 35% bro
pippo-ens6-9 worker localhost 12172 392M 280M 23% bro
pippo-ens6-10 worker localhost 12259 395M 286M 35% bro
pippo-ens6-11 worker localhost 12210 386M 275M 23% bro
pippo-ens6-12 worker localhost 12191 385M 274M 29% bro
pippo-ens6-13 worker localhost 12230 383M 273M 17% bro
pippo-ens6-14 worker localhost 12239 376M 274M 17% bro
pippo-ens6-15 worker localhost 12277 380M 278M 58% bro
pippo-ens6-16 worker localhost 12256 382M 273M 17% bro
pippo-ens6-17 worker localhost 12243 386M 274M 23% bro
pippo-ens6-18 worker localhost 12263 385M 274M 17% bro
pippo-ens6-19 worker localhost 12261 374M 271M 23% bro
pippo-ens6-20 worker localhost 12269 384M 274M 17% bro
pippo-ens6-21 worker localhost 12274 375M 273M 23% bro
pippo-ens6-22 worker localhost 12271 376M 275M 29% bro
mar 2 lug 2019, 09.45.11, UTC


root@pippo:/opt/bro/etc# broctl capstats; date

Interface kpps mbps (10s average)
----------------------------------------

localhost/af_packet::ens6 306.7 1709.8
mar 2 lug 2019, 09.45.49, UTC


pippo-ens6-1: 1562060771.800516 recvd=4440508 dropped=23660 link=4474631
pippo-ens6-2: 1562060771.829739 recvd=4954332 dropped=35587 link=4999273
pippo-ens6-3: 1562060771.833048 recvd=3143232 dropped=48827 link=3201663
pippo-ens6-4: 1562060771.846401 recvd=3159404 dropped=45611 link=3214578
pippo-ens6-5: 1562060771.856312 recvd=3945914 dropped=24086 link=3979340
pippo-ens6-6: 1562060771.878124 recvd=8025605 dropped=124340 link=8159688
pippo-ens6-7: 1562060771.887686 recvd=4834907 dropped=52358 link=4896547
pippo-ens6-8: 1562060771.921463 recvd=4326438 dropped=50134 link=4385838
pippo-ens6-9: 1562060771.933841 recvd=3872913 dropped=51439 link=3933736
pippo-ens6-10: 1562060771.944351 recvd=3283360 dropped=15324 link=3309114
pippo-ens6-11: 1562060771.954047 recvd=3817582 dropped=48991 link=3875976
pippo-ens6-12: 1562060771.974191 recvd=4309595 dropped=12384 link=4331976
pippo-ens6-13: 1562060771.992843 recvd=3174228 dropped=17002 link=3201665
pippo-ens6-14: 1562060772.013743 recvd=4818962 dropped=20005 link=4849634
pippo-ens6-15: 1562060772.018841 recvd=5139650 dropped=41132 link=5190180
pippo-ens6-16: 1562060772.035690 recvd=4494309 dropped=48128 link=4551691
pippo-ens6-17: 1562060772.051758 recvd=3331104 dropped=19635 link=3360929
pippo-ens6-18: 1562060772.056880 recvd=2996283 dropped=20789 link=3027647
pippo-ens6-19: 1562060772.066579 recvd=4617153 dropped=47387 link=4674400
pippo-ens6-20: 1562060772.078753 recvd=5006667 dropped=40328 link=5056872
pippo-ens6-21: 1562060772.090028 recvd=3354715 dropped=32459 link=3396762
pippo-ens6-22: 1562060772.094976 recvd=3041684 dropped=36703 link=3087745
mar 2 lug 2019, 09.46.12, UTC


sostat | less
...
=========================================================================
CPU Usage

=========================================================================
Load average for the last 1, 5, and 15 minutes:
7.40 10.05 10.68
Processing units: 24
...
=========================================================================
Packets received during last monitoring interval (600 seconds)
=========================================================================

ens6: 221668111

=========================================================================
Packet Loss Stats
=========================================================================

NIC:

ens6:

RX packets:45473495915 dropped:3587016 TX packets:0 dropped:0 -> 0,007%


I switched to pf_ring
(pf_ring)
# cat node.cfg
[manager]
type=manager
host=localhost

[logger]
type=logger

host=localhost

[proxy0]
type=proxy
host=localhost

[proxy1]
type=proxy
host=localhost

[proxy2]
type=proxy
host=localhost

[proxy3]
type=proxy
host=localhost

[proxy4]
type=proxy
host=localhost

[pippo-ens6]
type=worker
host=localhost
interface=ens6
lb_method=pf_ring
lb_procs=22
pin_cpus=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21


root@pippo:/opt/bro/etc# broctl top; date
Name Type Host Pid VSize Rss Cpu Cmd
logger logger localhost 24512 511M 125M 23% bro
manager manager localhost 24571 208M 106M 23% bro
proxy0 proxy localhost 24681 200M 98M 5% bro
proxy1 proxy localhost 24686 199M 97M 11% bro
proxy2 proxy localhost 24688 195M 93M 5% bro
proxy3 proxy localhost 24692 198M 96M 5% bro
proxy4 proxy localhost 24694 199M 96M 5% bro
pippo-ens6-1 worker localhost 24941 848M 736M 23% bro
pippo-ens6-2 worker localhost 24972 849M 743M 29% bro
pippo-ens6-3 worker localhost 24994 844M 740M 17% bro
pippo-ens6-4 worker localhost 24999 868M 765M 23% bro
pippo-ens6-5 worker localhost 25013 861M 750M 23% bro
pippo-ens6-6 worker localhost 25012 849M 736M 82% bro
pippo-ens6-7 worker localhost 25030 848M 738M 23% bro
pippo-ens6-8 worker localhost 25065 849M 737M 23% bro
pippo-ens6-9 worker localhost 25073 853M 739M 23% bro
pippo-ens6-10 worker localhost 25094 851M 738M 35% bro
pippo-ens6-11 worker localhost 25095 838M 734M 29% bro
pippo-ens6-12 worker localhost 25124 842M 739M 29% bro
pippo-ens6-13 worker localhost 25113 847M 736M 23% bro
pippo-ens6-14 worker localhost 25116 840M 735M 23% bro
pippo-ens6-15 worker localhost 25119 850M 737M 17% bro
pippo-ens6-16 worker localhost 25135 847M 740M 23% bro
pippo-ens6-17 worker localhost 25151 848M 735M 29% bro
pippo-ens6-18 worker localhost 25158 861M 749M 17% bro
pippo-ens6-19 worker localhost 25157 843M 739M 29% bro
pippo-ens6-20 worker localhost 25149 842M 738M 35% bro
pippo-ens6-21 worker localhost 25148 856M 743M 23% bro
pippo-ens6-22 worker localhost 25163 849M 737M 23% bro
mar 2 lug 2019, 10.00.12, UTC


root@pippo:/opt/bro/etc# broctl capstats; date

Interface kpps mbps (10s average)
----------------------------------------

localhost/ens6 310.0 1617.8
mar 2 lug 2019, 09.59.49, UTC


root@pippo:/opt/bro/etc# broctl netstats; date
pippo-ens6-1: 1562061544.183982 recvd=5527146 dropped=0 link=5527146
pippo-ens6-2: 1562061544.187125 recvd=7479342 dropped=0 link=7479342
pippo-ens6-3: 1562061544.206122 recvd=8386393 dropped=0 link=8386393
pippo-ens6-4: 1562061544.217965 recvd=8169776 dropped=0 link=8169776
pippo-ens6-5: 1562061544.248729 recvd=20660370 dropped=0 link=20660370
pippo-ens6-6: 1562061544.248565 recvd=10125690 dropped=0 link=10125690
pippo-ens6-7: 1562061544.273462 recvd=7080329 dropped=0 link=7080329
pippo-ens6-8: 1562061544.317390 recvd=5824115 dropped=0 link=5824115
pippo-ens6-9: 1562061544.331469 recvd=5850124 dropped=0 link=5850124
pippo-ens6-10: 1562061544.342647 recvd=9492890 dropped=0 link=9492890
pippo-ens6-11: 1562061544.367566 recvd=7433227 dropped=0 link=7433227
pippo-ens6-12: 1562061544.394588 recvd=6913040 dropped=0 link=6913040
pippo-ens6-13: 1562061544.406448 recvd=10151704 dropped=0 link=10151704
pippo-ens6-14: 1562061544.408390 recvd=9793967 dropped=0 link=9793967
pippo-ens6-15: 1562061544.417383 recvd=3962072 dropped=0 link=3962072
pippo-ens6-16: 1562061544.448472 recvd=10213614 dropped=0 link=10213614
pippo-ens6-17: 1562061544.469963 recvd=10737682 dropped=0 link=10737682
pippo-ens6-18: 1562061544.483759 recvd=13321027 dropped=0 link=13321027
pippo-ens6-19: 1562061544.481018 recvd=7178528 dropped=0 link=7178528
pippo-ens6-20: 1562061544.499316 recvd=7723821 dropped=0 link=7723821
pippo-ens6-21: 1562061544.515481 recvd=5384211 dropped=0 link=5384211
pippo-ens6-22: 1562061544.522227 recvd=18151047 dropped=0 link=18151047
mar 2 lug 2019, 09.59.04, UTC


sostat | less
...
=========================================================================
CPU Usage
=========================================================================
Load average for the last 1, 5, and 15 minutes:
9.87 9.16 9.68
Processing units: 24
...
=========================================================================
Packets received during last monitoring interval (600 seconds)
=========================================================================

ens6: 207993980

=========================================================================
Packet Loss Stats
=========================================================================

NIC:

ens6:

RX packets:45786909757 dropped:4197221 TX packets:0 dropped:0



In both cases drops are 0% but pf_ring case is a real 0%
Then I used the configuration you suggested me
(af_packet)
# cat node.cfg
[manager]
type=manager
host=localhost

[logger]
type=logger
host=localhost

[proxy0]
type=proxy
host=localhost

[pippo-ens6]
type=worker
host=localhost
interface=ens6
lb_method=custom
lb_procs=18
pin_cpus=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17
af_packet_fanout_id=31
af_packet_fanout_mode=AF_Packet::FANOUT_HASH
af_packet_buffer_size=128*1024*1024


root@pippo:/opt/bro/etc# broctl capstats; date

Interface kpps mbps (10s average)
----------------------------------------

localhost/af_packet::ens6 283.2 1561.7
mar 2 lug 2019, 11.21.21, UTC

root@pippo:/opt/bro/etc# broctl netstats; date
pippo-ens6-1: 1562066501.385745 recvd=1855677 dropped=5222 link=1869990
pippo-ens6-2: 1562066501.388894 recvd=1770735 dropped=4299 link=1784466
pippo-ens6-3: 1562066501.405727 recvd=2940992 dropped=5700 link=2956111
pippo-ens6-4: 1562066501.435440 recvd=7126180 dropped=12839 link=7149283
pippo-ens6-5: 1562066501.447823 recvd=3621441 dropped=8165 link=3639041
pippo-ens6-6: 1562066501.443317 recvd=5434103 dropped=36345 link=5479948
pippo-ens6-7: 1562066501.485827 recvd=2731612 dropped=8080 link=2749116
pippo-ens6-8: 1562066501.522599 recvd=1653049 dropped=19225 link=1682774
pippo-ens6-9: 1562066501.526536 recvd=3957844 dropped=17705 link=3985804
pippo-ens6-10: 1562066501.545781 recvd=2185032 dropped=17045 link=2212337
pippo-ens6-11: 1562066501.570669 recvd=2393858 dropped=18284 link=2422558
pippo-ens6-12: 1562066501.581712 recvd=2885739 dropped=14567 link=2910559
pippo-ens6-13: 1562066501.588965 recvd=1548741 dropped=18640 link=1577746
pippo-ens6-14: 1562066501.600686 recvd=2140053 dropped=19592 link=2169904
pippo-ens6-15: 1562066501.606776 recvd=1853946 dropped=18230 link=1882578
pippo-ens6-16: 1562066501.633491 recvd=2766598 dropped=18298 link=2795258
pippo-ens6-17: 1562066501.645098 recvd=2587777 dropped=18907 link=2617119
pippo-ens6-18: 1562066501.657072 recvd=2700856 dropped=19070 link=2730362
mar 2 lug 2019, 11.21.41, UTC

root@pippo:/opt/bro/etc# broctl top; date
Name Type Host Pid VSize Rss Cpu Cmd
logger logger localhost 8102 487M 112M 18% bro
manager manager localhost 8159 205M 103M 37% bro
proxy0 proxy localhost 8313 199M 97M 0% bro
pippo-ens6-1 worker localhost 8469 377M 276M 31% bro
pippo-ens6-2 worker localhost 8512 379M 277M 25% bro
pippo-ens6-3 worker localhost 8506 373M 271M 25% bro
pippo-ens6-4 worker localhost 8533 381M 271M 31% bro
pippo-ens6-5 worker localhost 8530 370M 269M 25% bro
pippo-ens6-6 worker localhost 8566 383M 272M 43% bro
pippo-ens6-7 worker localhost 8578 391M 274M 43% bro
pippo-ens6-8 worker localhost 8587 387M 276M 25% bro
pippo-ens6-9 worker localhost 8618 372M 270M 25% bro
pippo-ens6-10 worker localhost 8631 382M 272M 31% bro
pippo-ens6-11 worker localhost 8639 384M 273M 18% bro
pippo-ens6-12 worker localhost 8640 379M 269M 31% bro
pippo-ens6-13 worker localhost 8644 379M 270M 25% bro
pippo-ens6-14 worker localhost 8637 382M 272M 31% bro
pippo-ens6-15 worker localhost 8648 380M 270M 25% bro
pippo-ens6-16 worker localhost 8652 382M 272M 25% bro
pippo-ens6-17 worker localhost 8657 374M 273M 31% bro
pippo-ens6-18 worker localhost 8653 379M 268M 25% bro
mar 2 lug 2019, 11.22.02, UTC

sostat | less
....
=========================================================================
CPU Usage
=========================================================================
Load average for the last 1, 5, and 15 minutes:
7.60 7.12 7.47
Processing units: 24
...
=========================================================================
Packets received during last monitoring interval (600 seconds)
=========================================================================

ens6: 178795641

=========================================================================
Packet Loss Stats
=========================================================================

NIC:

ens6:

RX packets:47351238105 dropped:5213363 TX packets:0 dropped:0

-------------------------------------------------------------------------

pf_ring:
-------------------------------------------------------------------------

IDS Engine (suricata) packet drops:

/nsm/sensor_data/pippo-ens6/stats.log

No packet drops reported.

-------------------------------------------------------------------------

Bro:

Average packet loss as percent across all Bro workers: 0.367924

pippo-ens6-1: 1562066581.565780 recvd=2785636 dropped=5222 link=2799951
pippo-ens6-2: 1562066581.573782 recvd=2728230 dropped=4299 link=2741955
pippo-ens6-3: 1562066581.590463 recvd=3839321 dropped=5700 link=3854451
pippo-ens6-4: 1562066581.600505 recvd=10519492 dropped=12839 link=10542605
pippo-ens6-5: 1562066581.627729 recvd=5107920 dropped=8165 link=5125528
pippo-ens6-6: 1562066581.622252 recvd=7479859 dropped=36345 link=7526062
pippo-ens6-7: 1562066581.649114 recvd=4292222 dropped=8080 link=4309765
pippo-ens6-8: 1562066581.682612 recvd=2468734 dropped=19225 link=2498475
pippo-ens6-9: 1562066581.706439 recvd=5749731 dropped=17705 link=5777693
pippo-ens6-10: 1562066581.715829 recvd=3201321 dropped=17045 link=3228632
pippo-ens6-11: 1562066581.725962 recvd=3480037 dropped=18284 link=3508734
pippo-ens6-12: 1562066581.736620 recvd=4539343 dropped=14567 link=4564174
pippo-ens6-13: 1562066581.744135 recvd=2427048 dropped=18640 link=2456048
pippo-ens6-14: 1562066581.760831 recvd=3013499 dropped=19592 link=3043371
pippo-ens6-15: 1562066581.772103 recvd=2599670 dropped=18230 link=2628310
pippo-ens6-16: 1562066581.795470 recvd=3883006 dropped=18298 link=3911626
pippo-ens6-17: 1562066581.810055 recvd=3840911 dropped=18907 link=3870259
pippo-ens6-18: 1562066581.826959 recvd=3743481 dropped=19070 link=3772986

No capture loss reported.

-------------------------------------------------------------------------



In this case af_packet have 0.36% of drop
And now same configuration but with pf_ring
(pf_ring)
# cat node.cfg
[manager]
type=manager
host=localhost

[logger]
type=logger
host=localhost

[proxy0]
type=proxy
host=localhost

[pippo-ens6]
type=worker
host=localhost
interface=ens6
lb_method=pf_ring
lb_procs=18
pin_cpus=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17


root@pippo:/opt/bro/etc# broctl capstats; date

Interface kpps mbps (10s average)
----------------------------------------

localhost/ens6 269.1 1542.1
mar 2 lug 2019, 11.27.58, UTC

root@pippo:/opt/bro/etc# broctl netstats; date
pippo-ens6-1: 1562066896.740120 recvd=6546793 dropped=0 link=6546793
pippo-ens6-2: 1562066896.747094 recvd=2206942 dropped=0 link=2206942
pippo-ens6-3: 1562066896.754251 recvd=1739973 dropped=0 link=1739973
pippo-ens6-4: 1562066896.779290 recvd=2269913 dropped=0 link=2269913
pippo-ens6-5: 1562066896.794342 recvd=3460145 dropped=0 link=3460145
pippo-ens6-6: 1562066896.795470 recvd=3128399 dropped=0 link=3128399
pippo-ens6-7: 1562066896.823146 recvd=1700794 dropped=0 link=1700794
pippo-ens6-8: 1562066896.875335 recvd=3754045 dropped=0 link=3754045
pippo-ens6-9: 1562066896.897282 recvd=2346768 dropped=0 link=2346768
pippo-ens6-10: 1562066896.903137 recvd=2701147 dropped=0 link=2701147
pippo-ens6-11: 1562066896.911325 recvd=1907739 dropped=0 link=1907739
pippo-ens6-12: 1562066896.924124 recvd=2260111 dropped=0 link=2260111
pippo-ens6-13: 1562066896.927661 recvd=2259219 dropped=0 link=2259219
pippo-ens6-14: 1562066896.961993 recvd=2141611 dropped=0 link=2141611
pippo-ens6-15: 1562066896.968456 recvd=2125300 dropped=0 link=2125300
pippo-ens6-16: 1562066896.999332 recvd=1596994 dropped=0 link=1596994
pippo-ens6-17: 1562066897.012060 recvd=3203298 dropped=0 link=3203298
pippo-ens6-18: 1562066897.016508 recvd=1988023 dropped=0 link=1988023
mar 2 lug 2019, 11.28.17, UTC

root@pippo:/opt/bro/etc# broctl top; date
Name Type Host Pid VSize Rss Cpu Cmd
logger logger localhost 17931 491M 113M 25% bro
manager manager localhost 17994 205M 103M 18% bro
proxy0 proxy localhost 18047 199M 97M 6% bro
pippo-ens6-1 worker localhost 18244 838M 725M 37% bro
pippo-ens6-2 worker localhost 18218 825M 723M 31% bro
pippo-ens6-3 worker localhost 18250 830M 721M 12% bro
pippo-ens6-4 worker localhost 18262 833M 723M 18% bro
pippo-ens6-5 worker localhost 18266 833M 723M 25% bro
pippo-ens6-6 worker localhost 18293 820M 718M 62% bro
pippo-ens6-7 worker localhost 18302 835M 725M 12% bro
pippo-ens6-8 worker localhost 18310 822M 720M 25% bro
pippo-ens6-9 worker localhost 18329 832M 722M 18% bro
pippo-ens6-10 worker localhost 18351 822M 720M 43% bro
pippo-ens6-11 worker localhost 18348 832M 721M 25% bro
pippo-ens6-12 worker localhost 18347 837M 727M 31% bro
pippo-ens6-13 worker localhost 18365 832M 721M 31% bro
pippo-ens6-14 worker localhost 18364 846M 720M 25% bro
pippo-ens6-15 worker localhost 18370 826M 723M 18% bro
pippo-ens6-16 worker localhost 18372 829M 718M 25% bro
pippo-ens6-17 worker localhost 18374 830M 720M 25% bro
pippo-ens6-18 worker localhost 18380 832M 722M 25% bro
mar 2 lug 2019, 11.28.45, UTC


sostat | less
...
=========================================================================
CPU Usage
=========================================================================
Load average for the last 1, 5, and 15 minutes:
7.12 7.13 7.35
Processing units: 24
...
=========================================================================
Packets received during last monitoring interval (600 seconds)
=========================================================================

ens6: 183399975

=========================================================================
Packet Loss Stats
=========================================================================

NIC:

ens6:

RX packets:47461289142 dropped:5215584 TX packets:0 dropped:0

-------------------------------------------------------------------------

pf_ring:

Appl. Name: bro-ens6
Tot Packets: 2856559
Tot Pkt Lost: 0
Loss as a percentage: 0


Appl. Name: bro-ens6
Tot Packets: 10019654
Tot Pkt Lost: 0
Loss as a percentage: 0


Appl. Name: bro-ens6
Tot Packets: 2477546
Tot Pkt Lost: 0
Loss as a percentage: 0


Appl. Name: bro-ens6
Tot Packets: 3384410
Tot Pkt Lost: 0
Loss as a percentage: 0


Appl. Name: bro-ens6
Tot Packets: 4809556
Tot Pkt Lost: 0
Loss as a percentage: 0


Appl. Name: bro-ens6
Tot Packets: 4570909
Tot Pkt Lost: 0
Loss as a percentage: 0


Appl. Name: bro-ens6
Tot Packets: 2448733
Tot Pkt Lost: 0
Loss as a percentage: 0


Appl. Name: bro-ens6
Tot Packets: 5272143
Tot Pkt Lost: 0
Loss as a percentage: 0


Appl. Name: bro-ens6
Tot Packets: 3185613
Tot Pkt Lost: 0
Loss as a percentage: 0


Appl. Name: bro-ens6
Tot Packets: 3182650
Tot Pkt Lost: 0
Loss as a percentage: 0


Appl. Name: bro-ens6
Tot Packets: 2555882
Tot Pkt Lost: 0
Loss as a percentage: 0


Appl. Name: bro-ens6
Tot Packets: 3835354
Tot Pkt Lost: 0
Loss as a percentage: 0


Appl. Name: bro-ens6
Tot Packets: 3156294
Tot Pkt Lost: 0
Loss as a percentage: 0


Appl. Name: bro-ens6
Tot Packets: 3641179
Tot Pkt Lost: 0
Loss as a percentage: 0


Appl. Name: bro-ens6
Tot Packets: 2840170
Tot Pkt Lost: 0
Loss as a percentage: 0


Appl. Name: bro-ens6
Tot Packets: 2246308
Tot Pkt Lost: 0
:


Appl. Name: bro-ens6
Tot Packets: 4467650
Tot Pkt Lost: 0
Loss as a percentage: 0


Appl. Name: bro-ens6
Tot Packets: 2934095
Tot Pkt Lost: 0
Loss as a percentage: 0

-------------------------------------------------------------------------

IDS Engine (suricata) packet drops:

/nsm/sensor_data/pippo-ens6/stats.log

No packet drops reported.

-------------------------------------------------------------------------

Bro:

Average packet loss as percent across all Bro workers: 0.000000

pippo-ens6-1: 1562066966.330195 recvd=10053840 dropped=0 link=10053840
pippo-ens6-2: 1562066966.337055 recvd=2861641 dropped=0 link=2861641
pippo-ens6-3: 1562066966.344287 recvd=2484816 dropped=0 link=2484816
pippo-ens6-4: 1562066966.352003 recvd=3389116 dropped=0 link=3389116
pippo-ens6-5: 1562066966.364295 recvd=4817132 dropped=0 link=4817132
pippo-ens6-6: 1562066966.378531 recvd=4575882 dropped=0 link=4575882
pippo-ens6-7: 1562066966.397703 recvd=2452376 dropped=0 link=2452376
pippo-ens6-8: 1562066966.420244 recvd=5286136 dropped=0 link=5286136
pippo-ens6-9: 1562066966.427270 recvd=3190821 dropped=0 link=3190821
pippo-ens6-10: 1562066966.458195 recvd=3844846 dropped=0 link=3844846
pippo-ens6-11: 1562066966.476192 recvd=2559486 dropped=0 link=2559486
pippo-ens6-12: 1562066966.484009 recvd=3187907 dropped=0 link=3187907
pippo-ens6-13: 1562066966.507566 recvd=3651283 dropped=0 link=3651283
pippo-ens6-14: 1562066966.517062 recvd=3167416 dropped=0 link=3167416
pippo-ens6-15: 1562066966.538441 recvd=2851031 dropped=0 link=2851031
pippo-ens6-16: 1562066966.559338 recvd=2252386 dropped=0 link=2252386
pippo-ens6-17: 1562066966.562407 recvd=4481435 dropped=0 link=4481435
pippo-ens6-18: 1562066966.581381 recvd=2945829 dropped=0 link=2945829

No capture loss reported.

-------------------------------------------------------------------------


As you can see pf_ring is real 0% af_packet not.
Now is summer so we have a low traffic situation.
If you have other suggestions, let's continue with the tests :-).

I had already read the papers you suggested me, they are very interesting.
Those papers refer to suricata and af_packet. They include very deep changes to the system and I don't think I'll ever do them especially if I can get the same performance using pf_ring and without modifying anything.

What I think, from the perspective of open source, to close the door to a solution over another is wrong, especially if the behavior varies as much from one hardware to another. On security onion both can coexist, obviously the decision is up to you.

Now let me suggest you some readings, on the subject of pf_ring:
https://www.ntop.org/ntop/introducing-nprobe-agent-packetless-system-introspected-network-visibility/
https://www.ntop.org/ntop/system-introspected-network-and-container-visibility-a-quick-start-guide/
https://www.ntop.org/pf_ring/introducing-pf_ring-configuration-wizard/

Thank you very much Doug

See you soon

Simone

[Doug Burks]

Hi Simone,

Replies inline.

Were you running into any particular issue that caused you to increase your number of proxies to 5?  The LBL paper recommends 5 proxies for 100Gbps, but I wouldn't think that you would need 5 proxies for 2-4Gbps. 

Do these dropped numbers increase over time or stay the same?  If they stay roughly the same, could it just be drops when Bro and/or its AF_PACKET plugin is initializing?

Are you referring to this line?

RX packets:45786909757 dropped:4197221 TX packets:0 dropped:0

In this case, we do see the NIC reporting packet drops while you're running PF_RING.  Have you noticed this before?  What MTU is your NIC set to?  Are you monitoring any jumbo frame traffic?

The information you provided above gives us some good insight to your system and the traffic it's monitoring, so thanks for that!  It might also be helpful if you could send the entire sostat-redacted output. Additionally, it might be good to see the output of "ethtool -S" on your sniffing interface.

Based on some of my preliminary comments above, I wonder if what you are seeing may be attributed to one or more of the following:

- packet loss during initialization of Bro and/or it's AF_PACKET plugin

- differences in the way that Bro reports dropped packets when consuming via PF_RING vs the AF_PACKET plugin

- jumbo frame traffic hitting a NIC with non-jumbo MTU

I had already read the papers you suggested me, they are very interesting.
Those papers refer to suricata and af_packet. They include very deep changes to the system and I don't think I'll ever do them especially if I can get the same performance using pf_ring and without modifying anything.

I understand.  I wasn't saying that you had to make all of those changes, I was simply pointing out that if AF_PACKET can handle 20Gbps, it stands to reason that it should be able to handle 4Gbps fairly easily.  :)

 

What I think, from the perspective of open source, to close the door to a solution over another is wrong, especially if the behavior varies as much from one hardware to another. On security onion both can coexist, obviously the decision is up to you.

Now let me suggest you some readings, on the subject of pf_ring:
https://www.ntop.org/ntop/introducing-nprobe-agent-packetless-system-introspected-network-visibility/
https://www.ntop.org/ntop/system-introspected-network-and-container-visibility-a-quick-start-guide/
https://www.ntop.org/pf_ring/introducing-pf_ring-configuration-wizard/

Yes, I'm subscribed to the ntop RSS feed and so I read those articles when they were published.  Nprobe Agent seems interesting, but it looks like it's not open source so we wouldn't be able to leverage it in Security Onion anyway.

 

Thank you very much Doug

Thanks, Simone!  :)

 

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/2df517de-6834-4a9e-90ce-ecf343dedcc6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Simone Bonetti]

Hi Doug,
replies inline.

Il giorno martedì 2 luglio 2019 23:48:20 UTC+2, Doug Burks ha scritto:
>
> Were you running into any particular issue that caused you to increase your number of proxies to 5?  The LBL paper recommends 5 proxies for 100Gbps, but I wouldn't think that you would need 5 proxies for 2-4Gbps. 

Not now, but in high traffic situation could be (proxy more than one). I'll test that in future. For now I'll use only a proxy but I increase workers number from 18 to 22 (As we saw it's useful only for af_packet)

Just to be clear, this is the configuration in test for af_packet:

[manager]
type=manager
host=localhost

[logger]
type=logger
host=localhost

[proxy0]
type=proxy
host=localhost

[pippo-ens6]
type=worker
host=localhost
interface=ens6

#lb_method=pf_ring

lb_method=custom
lb_procs=22
pin_cpus=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21
# Optional parameters for per node configuration:
af_packet_fanout_id=23
af_packet_fanout_mode=AF_Packet::FANOUT_HASH
af_packet_buffer_size=128*1024*1024

and this is for pf_ring:

[manager]
type=manager
host=localhost

[logger]
type=logger
host=localhost

[proxy0]
type=proxy
host=localhost

[pippo-ens6]
type=worker
host=localhost
interface=ens6
lb_method=pf_ring

lb_procs=22
pin_cpus=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21

> Do these dropped numbers increase over time or stay the same?  If they stay roughly the same, could it just be drops when Bro and/or its AF_PACKET plugin is initializing?

I think they stay the same. I don't think it's a AF_PACKET initialization. You can see that in attachment.

> Are you referring to this line?
> RX packets:45786909757 dropped:4197221 TX packets:0 dropped:0
>
>
> In this case, we do see the NIC reporting packet drops while you're running PF_RING.  Have you noticed this before?  What MTU is your NIC set to?  Are you monitoring any jumbo frame traffic?

NIC reporting packet drops in both situations, you can see that. I think this is an issue about hardware. As you can see the drops are very very low.

sosetup defined NIC settings I add only rss=1 as you suggest here: https://github.com/Security-Onion-Solutions/security-onion/wiki/PostInstallation
I see now there isn't a reference to rss in the new documents: https://github.com/Security-Onion-Solutions/security-onion/wiki/PostInstallation

> The information you provided above gives us some good insight to your system and the traffic it's monitoring, so thanks for that!  It might also be helpful if you could send the entire sostat-redacted output. Additionally, it might be good to see the output of "ethtool -S" on your sniffing interface.

In attachment as you asked

> Based on some of my preliminary comments above, I wonder if what you are seeing may be attributed to one or more of the following:
> - packet loss during initialization of Bro and/or it's AF_PACKET plugin

It's not the case, however broctl doesn't show drops during pf_ring startup/initialization

> - differences in the way that Bro reports dropped packets when consuming via PF_RING vs the AF_PACKET plugin

It could be. Do you have any information about that?

> - jumbo frame traffic hitting a NIC with non-jumbo MTU

Tell me how to do for this
However this is the same for af_packet and pf_ring


Thanks Doug :)


Simone

[Doug Burks]

Hi Simone,

Replies inline.

If I'm reading it correctly, your first afpacket sostat at 201907031210 shows:

SO-server-ens6-1: 1562148661.586952 recvd=2298214 dropped=26784 link=2333921
SO-server-ens6-2: 1562148661.592544 recvd=2387292 dropped=23200 link=2419376
SO-server-ens6-3: 1562148661.603563 recvd=4546621 dropped=21036 link=4576707
SO-server-ens6-4: 1562148661.617930 recvd=2412645 dropped=21847 link=2443490
SO-server-ens6-5: 1562148661.617624 recvd=2729953 dropped=27272 link=2765935
SO-server-ens6-6: 1562148661.642729 recvd=1391457 dropped=42251 link=1442540
SO-server-ens6-7: 1562148661.640497 recvd=2141619 dropped=35824 link=2186298
SO-server-ens6-8: 1562148661.673099 recvd=1617977 dropped=33920 link=1660812
SO-server-ens6-9: 1562148661.680403 recvd=2051842 dropped=33148 link=2094284
SO-server-ens6-10: 1562148661.696060 recvd=1933332 dropped=33939 link=1977291
SO-server-ens6-11: 1562148661.702966 recvd=2244068 dropped=32885 link=2287246
SO-server-ens6-12: 1562148661.713366 recvd=2011593 dropped=31819 link=2053311
SO-server-ens6-13: 1562148661.725283 recvd=3395898 dropped=33371 link=3438960
SO-server-ens6-14: 1562148661.731821 recvd=3762135 dropped=33077 link=3805295
SO-server-ens6-15: 1562148661.742782 recvd=1466969 dropped=32765 link=1510004
SO-server-ens6-16: 1562148661.777132 recvd=1756220 dropped=33228 link=1800723
SO-server-ens6-17: 1562148661.786974 recvd=3882214 dropped=33970 link=3926531
SO-server-ens6-18: 1562148661.796399 recvd=2533313 dropped=33720 link=2577308
SO-server-ens6-19: 1562148661.800096 recvd=3491171 dropped=32812 link=3534402
SO-server-ens6-20: 1562148661.816829 recvd=2631226 dropped=33488 link=2675071
SO-server-ens6-21: 1562148661.820669 recvd=2482636 dropped=34380 link=2527404
SO-server-ens6-22: 1562148661.822687 recvd=3738825 dropped=34975 link=3784338

and then your second afpacket sostat at 201907031321 shows:

SO-server-ens6-1: 1562152898.316351 recvd=49243898 dropped=26784 link=49279608
SO-server-ens6-2: 1562152898.322547 recvd=63366602 dropped=23200 link=63398697
SO-server-ens6-3: 1562152898.328442 recvd=91229374 dropped=21036 link=91259459
SO-server-ens6-4: 1562152898.342688 recvd=53175392 dropped=21847 link=53206232
SO-server-ens6-5: 1562152898.352773 recvd=50601095 dropped=27272 link=50637080
SO-server-ens6-6: 1562152898.353884 recvd=35797779 dropped=42251 link=35848816
SO-server-ens6-7: 1562152898.368780 recvd=47555386 dropped=35824 link=47600035
SO-server-ens6-8: 1562152898.428704 recvd=46307176 dropped=33920 link=46349915
SO-server-ens6-9: 1562152898.432010 recvd=56843246 dropped=33148 link=56885674
SO-server-ens6-10: 1562152898.428814 recvd=53495243 dropped=33939 link=53539335
SO-server-ens6-11: 1562152898.448095 recvd=52049848 dropped=32885 link=52093035
SO-server-ens6-12: 1562152898.459745 recvd=62154282 dropped=31819 link=62195978
SO-server-ens6-13: 1562152898.465291 recvd=66812983 dropped=33371 link=66856043
SO-server-ens6-14: 1562152898.476691 recvd=65450915 dropped=33077 link=65494072
SO-server-ens6-15: 1562152898.488086 recvd=37176658 dropped=32765 link=37219678
SO-server-ens6-16: 1562152898.510211 recvd=49193274 dropped=33228 link=49236817
SO-server-ens6-17: 1562152898.516788 recvd=60893179 dropped=33970 link=60937511
SO-server-ens6-18: 1562152898.531362 recvd=55963889 dropped=33720 link=56007881
SO-server-ens6-19: 1562152898.540671 recvd=60492017 dropped=32812 link=60535101
SO-server-ens6-20: 1562152898.546958 recvd=50302294 dropped=33488 link=50346147
SO-server-ens6-21: 1562152898.555795 recvd=47436481 dropped=34380 link=47481242
SO-server-ens6-22: 1562152898.566815 recvd=57381482 dropped=34975 link=57426850

So comparing those two, the recvd counts have increased as expected, but the dropped counts have stayed *exactly* the same.  This sounds like the dropped packets only occur at initialization and once fully initialized, there are no additional drops.

> Are you referring to this line?
> RX packets:45786909757 dropped:4197221 TX packets:0 dropped:0
>
>
> In this case, we do see the NIC reporting packet drops while you're running PF_RING.  Have you noticed this before?  What MTU is your NIC set to?  Are you monitoring any jumbo frame traffic?

NIC reporting packet drops in both situations, you can see that. I think this is an issue about hardware. As you can see the drops are very very low. 

sosetup defined NIC settings I add only rss=1 as you suggest here: https://github.com/Security-Onion-Solutions/security-onion/wiki/PostInstallation
I see now there isn't a reference to rss in the new documents: https://github.com/Security-Onion-Solutions/security-onion/wiki/PostInstallation

That rss setting has been moved to a different page in the new documentation:

https://securityonion.readthedocs.io/en/latest/performance.html#rss

 

> The information you provided above gives us some good insight to your system and the traffic it's monitoring, so thanks for that!  It might also be helpful if you could send the entire sostat-redacted output. Additionally, it might be good to see the output of "ethtool -S" on your sniffing interface.

In attachment as you asked


> Based on some of my preliminary comments above, I wonder if what you are seeing may be attributed to one or more of the following:
> - packet loss during initialization of Bro and/or it's AF_PACKET plugin
It's not the case, however broctl doesn't show drops during pf_ring startup/initialization

It sounds like Bro initialization is different when Bro is configured with lb_method=pf_ring vs using the af_packet plugin.

> - differences in the way that Bro reports dropped packets when consuming via PF_RING vs the AF_PACKET plugin
It could be. Do you have any information about that?

Perhaps based on the above, Bro with lb_method=pf_ring doesn't "see" packets until fully initialized and thus 0 drops.  Perhaps with the AF_PACKET plugin, Bro "sees" packets before fully initialized and since it is not fully initialized, it calls them "dropped"?

> - jumbo frame traffic hitting a NIC with non-jumbo MTU
Tell me how to do for this
However this is the same for af_packet and pf_ring

If you are monitoring traffic with jumbo frames, then you may need to increase the MTU on your sniffing interface:

https://securityonion.readthedocs.io/en/latest/production-deployment.html?highlight=mtu#setup

 

Thanks Doug :)

Thanks again Simone!