pf_ring packet drops - need help tuning

[Konrad W]

Hello,

I have been tuning one of my busier sensors and I need some help at this point. I am monitoring 6 1GE interfaces on this sensor and there are 2 interfaces that I keep seeing packet loss on pf_ring stats Snort cluster mostly. I since increased the pf_ring buffer size to 2GB for those and have 5 Bro workers on each and 5 Snort processes. I have about 8K SNORT rules enabled. I do not see much drops on Bro netstats and also IDS Engine (snort) packet drops are 0, however as you can see below pf_ring stats show lost packets for the 2 interfaces. The 2 interfaces in question, I expect the traffic to spike to 1Gbps at times and perhaps more therefore I do see some overruns on the interfaces itself. I know that is probably another issue I need to deal with but regardless of that, I would like to get rid of the pf_ring drops first. What other options do I have? Should I keep increasing the pf_ring buffer size and adding more workers to those interface until I hit the sweet spot or is there something else I need to look at this point?

My sensor has 128GB of RAM and 10core/20 thread intel xeon cpu. I have 16TB storage for captures on it in RAID 5 configuration.

See below some stats from the box and in regards to the 2 interfaces in question.

Thanks

Konrad



free -g
total used free shared buffers cached
Mem: 125 125 0 0 0 103
-/+ buffers/cache: 21 104
Swap: 92 0 92

top - 17:59:30 up 5 days, 15:22, 1 user, load average: 5.80, 6.99, 7.50
Tasks: 503 total, 15 running, 488 sleeping, 0 stopped, 0 zombie
Cpu(s): 8.9%us, 21.7%sy, 2.5%ni, 64.3%id, 0.0%wa, 0.0%hi, 2.5%si, 0.0%st
Mem: 131997616k total, 131327916k used, 669700k free, 370648k buffers
Swap: 97248464k total, 553904k used, 96694560k free, 107861992k cached

eth8 Link encap:Ethernet HWaddr ac:16:2d:9e:ed:a4
UP BROADCAST RUNNING NOARP PROMISC MULTICAST MTU:1500 Metric:1
RX packets:3510000488 errors:0 dropped:0 overruns:459558 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:713818426793 (713.8 GB) TX bytes:0 (0.0 B)
Memory:fbf80000-fc000000

eth9 Link encap:Ethernet HWaddr ac:16:2d:9e:ed:a5
UP BROADCAST RUNNING NOARP PROMISC MULTICAST MTU:1500 Metric:1
RX packets:9507333838 errors:0 dropped:0 overruns:11806791 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4912178627772 (4.9 TB) TX bytes:0 (0.0 B)
Memory:fbe80000-fbf00000

=========================================================================
Bro netstats
=========================================================================
Average packet loss as percent across all Bro workers: 0.000000

xxx-eth8-1: 1423061011.302026 recvd=6412462 dropped=0 link=6412462
xxx-eth8-2: 1423061011.503086 recvd=7201378 dropped=0 link=7201378
xxx-eth8-3: 1423061011.706895 recvd=7743506 dropped=0 link=7743506
xxx-eth8-4: 1423061011.906852 recvd=8096724 dropped=0 link=8096724
xxx-eth8-5: 1423061012.106019 recvd=7700595 dropped=0 link=7700595
xxx-eth9-1: 1423061012.307132 recvd=115235151 dropped=0 link=115235151
xxx-eth9-2: 1423061012.507643 recvd=93990988 dropped=1 link=93990988
xxx-eth9-3: 1423061012.707203 recvd=83839232 dropped=0 link=83839232
xxx-eth9-4: 1423061012.904978 recvd=95240640 dropped=0 link=95240640
xxx-eth9-5: 1423061013.105186 recvd=86131351 dropped=0 link=86131351

=========================================================================
IDS Engine (snort) packet drops
=========================================================================
/nsm/sensor_data/xxx-eth8/snort-1.stats last reported pkt_drop_percent as 0.000
/nsm/sensor_data/xxx-eth8/snort-2.stats last reported pkt_drop_percent as 0.000
/nsm/sensor_data/xxx-eth8/snort-3.stats last reported pkt_drop_percent as 0.000
/nsm/sensor_data/xxx-eth8/snort-4.stats last reported pkt_drop_percent as 0.000
/nsm/sensor_data/xxx-eth8/snort-5.stats last reported pkt_drop_percent as 0.000
/nsm/sensor_data/xxx-eth9/snort-1.stats last reported pkt_drop_percent as 0.000
/nsm/sensor_data/xxx-eth9/snort-2.stats last reported pkt_drop_percent as 0.000
/nsm/sensor_data/xxx-eth9/snort-3.stats last reported pkt_drop_percent as 0.000
/nsm/sensor_data/xxx-eth9/snort-4.stats last reported pkt_drop_percent as 0.000
/nsm/sensor_data/xxx-eth9/snort-5.stats last reported pkt_drop_percent as 0.000

/proc/net/pf_ring/12550-eth8.8287
Appl. Name : snort-cluster-61-socket-0
Tot Packets : 6411624
Tot Pkt Lost : 75714
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 4098
Num Free Slots : 4052

/proc/net/pf_ring/12568-eth8.8288
Appl. Name : snort-cluster-61-socket-0
Tot Packets : 7194633
Tot Pkt Lost : 77237
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 4098
Num Free Slots : 4095

/proc/net/pf_ring/12586-eth8.8290
Appl. Name : snort-cluster-61-socket-0
Tot Packets : 7727537
Tot Pkt Lost : 37541
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 4098
Num Free Slots : 4057

/proc/net/pf_ring/12605-eth8.8291
Appl. Name : snort-cluster-61-socket-0
Tot Packets : 8084803
Tot Pkt Lost : 51741
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 4098
Num Free Slots : 4085

/proc/net/pf_ring/12623-eth8.8289
Appl. Name : snort-cluster-61-socket-0
Tot Packets : 7691775
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 4098
Num Free Slots : 4064

/proc/net/pf_ring/13010-eth9.8295
Appl. Name : snort-cluster-62-socket-0
Tot Packets : 85929083
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 4098
Num Free Slots : 4038

/proc/net/pf_ring/13089-eth9.8296
Appl. Name : snort-cluster-62-socket-0
Tot Packets : 93804710
Tot Pkt Lost : 28308
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 4098
Num Free Slots : 3980

/proc/net/pf_ring/13111-eth9.8294
Appl. Name : snort-cluster-62-socket-0
Tot Packets : 95059450
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 4098
Num Free Slots : 4020

/proc/net/pf_ring/13130-eth9.8292
Appl. Name : snort-cluster-62-socket-0
Tot Packets : 83729455
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 4098
Num Free Slots : 4008

/proc/net/pf_ring/13148-eth9.8293
Appl. Name : snort-cluster-62-socket-0
Tot Packets : 114975178
Tot Pkt Lost : 0
TX: Send Errors : 0
Reflect: Fwd Errors: 0
Min Num Slots : 4098
Num Free Slots : 4014

[Doug Burks]

Have you tried increasing min_num_slots in
/etc/modprobe.d/pf_ring.conf and rebooting?

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com

[Konrad W]

Hey Doug,

No I did not try that. What should I set it to? Double it or more?

Thanks,

Konrad

[Doug Burks]

Try setting it to 65534, reboot, and see if that helps.

[Konrad W]

Thank you very much again. I have set this to 65534. Will check it daily if improved.

[stardj dre]

Hello Konrad,
I'm working on a similar setup with Snort and Bro (Bro control, pf ring zc, zbalance_ipc on an Ubuntu server with 64 GB RAM).
Snort isn't dropping packages, but Bro is, even at a low load with a 100-300 Mbps. Do you have any ideas on how to minimize the number of dropped packages in Bro?
Thanks in advance,
Andreas

[Doug Burks]

Hi Andreas,

It sounds like you're not running Security Onion, but instead manually
compiling Snort and Bro on your own Ubuntu installation. Support for
non-Security-Onion systems is beyond the scope of this Security Onion
group.


--
Doug Burks