SMTP traffic taking longer than other traffic types to appear on the Moloch UI

[Gareth Bromley]

We are beginning to notice that SMTP traffic appears to take much longer to get ingested into Moloch than other protocols e.g. http. In some cases it can take 5-15 minutes from an IDS alert being raised to be able to see the session in Moloch and download the PCAP for it, which we are not seeing for other traffic types.

We are not seeing anything obvious in the logs for Moloch, and hunting around obvious Linux performance elements shows no obvious issues. Is there any information I can pull from parser logs (if they exist) to understand why it may take so long?

Cheers

Gareth

[Andy]

5-15min is pretty normal for any type of session to show up, Moloch isn't a realtime system, there are lot of buffers/queues.  That said it mostly shouldn't matter what type of data it is.  Just remember that moloch isn't writing a SPI record until the session is "done", while your IDS is probably sending the alert as soon as it sees something bad.

For example if you look at your sessions tab at the graph, the last ~10 minutes will always be lower, and if you refresh the page you will notice the graph goes up for those minutes as they move out of being the last 10 minutes.

This is because things aren't written to ES unless 

* the tcp session closed and 5 seconds go by

* at least 10000 (maxPackets) have gone by

* 600 second (tcpTimeout) of idle communication

* 720 seconds (tcpSaveTimeout) since last write

* Any WISE queries finish (which if you have a slow email source MIGHT be the issue, but probably not)

To confirm this, find one of the email sessions in moloch that is "slow" and look at the start/stop time in moloch vs the pcap data, and it should fit one of the items above.  (If I had to guess its either a long lived SMTP sessions with lots of mail, or the other side isn't actually closing the connection and a timeout hits.)  If not I would love to get a sample pcap.