TCP ACKed unseen segment errors

[anup katariya]

Hi,

I recently setup moloch in our production and still learning.  We have number of taps connected to broker switch and then traffic is offloaded to one capture machine. I have two interface where traffic is split 50-50. 

I see below errors randomly in wireshark when I download some pcaps.  What does it mean? 

Moloch version = 1.0.0-1

Setup is single moloch capture process running with following config.

interface=eth2;eth3

magicMode=basic

pcapReadMethod=tpacketv3

tpacketv3NumThreads=2

pcapWriteMethod=simple

pcapWriteSize = 2560000

packetThreads=5

maxPacketsInQueue = 200000

We get around 2 TB of data a day. 

[anup katariya]

Attaching wireshark image again.

[Andy]

I don't see the images, but there are lots of google results for the subject text if thats what it is about.

[anup katariya]

Thanks Andy. 

So I understand that this means packets were either not captured by capture process or some issue when I download pcap file from viewer.   What could be reason? I don't see any errors in capture.log 

Could it be because we are splitting traffic between two interfaces?

Anup

[Andy]

This isnt a moloch issue, you wouldn't see anything in the capture logs. If your NPB isn't using symmetric hashing, or if you aren't load balancing based on the 5 tuple that would be the issue.  If you are using SPAN ports and the devices are overloaded or dropping packets that could be the issue.  If using vlans and SPANs make sure all the vlans are configured.  If you have asymmetric routing that could be the issue.  There are basically endless reasons.

Thanks,

Andy

[suraj....@integral.com]

So, we checked with our PBR vendor and are to understand that load-balancing is done on the following fields:

Source MAC address hashing for non-IP packets is ON
Destination MAC address hashing for non-IP packets is ON
Ethernet type hashing for non-IP packets is ON
VLAN ID hashing for non-IP packets is ON
VLAN priority hashing for non-IP packets is ON
Source MAC address hashing for IP packets is ON
Destination MAC address hashing for IP packets is ON
Ethernet type hashing for IP packets is ON
VLAN ID hashing for IP packets is ON
VLAN priority hashing for IP packets is ON
IP source address hashing is ON
IP destination address hashing is ON
IP protocol field hashing is ON
TCP/UDP source port hashing is ON
TCP/UDP destination port hashing is ON

Also, we aren't using SPAN ports (we use fiber TAPs), and they trickle down two 10Gbps interfaces (the aggregate bandwidth of the data received by the moloch machine is a maximum of 1Gbps during peek hours).

You are right in that the PBR needs symmetric hashing explicitly turned on. However, despite us turning off the load-balancing (to the moloch machine), and sending the machine all data down one interface, we see wireshark complain of "unseen segments" (and not just during peak hours).

It isn't the asymmetric routing or the VLANs, but could it be the cards on the machine that is causing this? We are using solarflare cards with the following driver and firmware version:

driver: sfc
version: 4.13.1.1034
firmware-version: 6.2.7.1000 rx1 tx1
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: yes

We have also configured the NICs as per the section "Network Card Config" in:

https://github.com/aol/moloch/wiki/FAQ#why-am-i-dropping-packets

Any help, would be greatly appreciated.

Regards,

suraj.

[Andy]

I would suggest you change your NPB to only hash on the 5 tuple.

I know nothing about solarflare.

Are you sure you aren't seeing any interface or moloch drops?

If you think its a moloch problem I would suggest using tcpdump to write to a file and then loading that into wireshark and/or moloch.