security onion not receiving SPAN traffic but wireshark does

[Prashant Shrivastva]

Dear Experts,


I have a Security Onion Master and Security Onion Sensor (where SPAN is going in).

On the Sensor, I run "tcpdump" command and no SPAN traffic is being received on the sniffing interface configured during setup.
Also there are no external traffic (SPAN traffic) in the Bro logs, conn.log

However when I unplug the SPAN cable and put in my laptop having wireshark, I RECEIVE the SPAN TRAFFIC. Which clearly indicates that the Mirroring Port configurations are correct.

What might be blocking the IDS to receive the SPAN traffic?

FYI - I am using Surricata rules.

Also I am attaching "sostat-redacted" output with this post.

Would be great if someone could help, have been breaking my head without any progress. Thanks

Regards
Prashant

[Wes Lambert]

I'm not sure I understand.

Based on your sostat output (PF_RING stats, etc) and the interface statistics, it looks like you are receiving traffic as intended.

Could you be using VLAN tags or some type of encapsulation?

How are you attending to run tcpdump against the interface?

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Prashant Shrivastva]

Hi Wes,

I am running tcpdump command as below:

# tcpdump -i ens160

ens160 is SPAN interface. I also ran tcpdump -i ens192 , and ens192 is management interface.

I only receive Broadcast and internal securityonion traffic in the tcpdump commands.

However when i plug off the SPAN cable from server (Esxi) and plung my laptop, then wireshark shows all the traffic including external and inbound.

Also there is no traffic apart from Broadcast and internal security onion traffic in my Bro logs and neither any thing gets fired in sguil database.

Thanks

Prashant

--

Thanks
with regards,
Prashant Shrivastva
8050674842

[Brian Dorr]

Can I ask what type of hardware is your sensor on? Standalone? Are you using virtual at all?

[Prashant Shrivastva]

Hi , 

It’s a Dell hardware , installed with ESXi and a virtual server is hosting my Security onion sensor . 

Now, I have configured a virtual switch which maps the physical adapter with the virual interface .

They are mapped to two virtual interface which is shown in the sensor VM . 

The sensor was working perfectly fine , but i had to reinstall the sensor as there was a data center activity. 

Thanks 

Prashant 

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

Thanks
with regards,
Prashant Shrivastva
8050674842

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Brian Dorr]

I would double check the ports you have set up ass pass through, based off everything you said, this seems to be where the issue exists. 

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

Thanks
with regards,
Prashant Shrivastva
8050674842

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Thanks
with regards,
Prashant Shrivastva
8050674842

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.

[ledin...@gmail.com]

Hi Prashant,

Be sure to check the MAC address setting for the SPAN interface. There are some key ESXi settings that will prevent traffic from flowing as expected. I will dig into my notes tomorrow morning to provide more details if you haven’t got this resolved by then.

[Prashant Shrivastva]

Hi Ledingtech,

Please let me know more details about this. I am really not sure what MAC address settings for SPAN need to be checked. Thanks

Regards

Prashant

On Sat, Jan 26, 2019 at 1:20 PM <ledin...@gmail.com> wrote:

Hi Prashant,

Be sure to check the MAC address setting for the SPAN interface.  There are some key ESXi settings that will prevent traffic from flowing as expected.  I will dig into my notes tomorrow morning to provide more details if you haven’t got this resolved by then.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[ledin...@gmail.com]

Good morning Prashant,

The key thing is to ensure promiscuous mode is enabled. I believe there was also one other setting that I had to modify but it escapes me at present but I will keep digging...

https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.security.doc/GUID-92F3AB1F-B4C5-4F25-A010-8820D7250350.html

[ledin...@gmail.com]

Hi Prashant,

I checked my notes and the other feature I was thinking of is "Allow forged transmits" but this would not be relevant for a SPAN port capturing traffic.

So I think the promiscuous mode setting might be part of the issue...

[Prashant Shrivastva]

Thank you for reply . But promiscuous mode enable was the first thing I checked . 

Promiscuous mode is enabled on the virtual adapter as well as on the virtual switch . 

Can you let me know how to identify which is SPAN port from the Security onion vm ? 

Also I am thinking to rebuild the virtual switch configuration at ESXi level . Can you give me a guide to build vswitch please ? 

Regards 

Prashant 

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Prashant Shrivastva]

Hi,

So I checked and confirmed my SPAN port from Esxi, there was a discovered subnet on one of the virtual interfaces, while the other had not discovered any subnets.

I am choosing the right SPAN interface and giving the same during the setup of interfaces in SO.

The weird part is, both my interfaces (management and SPAN) show the same traffic when tcpdump is run. Both show broadcast and internal security onion master and sensor traffic.

Kindly give a solution, I have been reinstalling SO and getting no where with this one. Any settings on virtual environment?

Regards

Prashant

[Prashant Shrivastva]

Also I would add one more information,

I am not able to edit the /etc/network/interfaces file.

It says the file is readonly file. I tried remounting the partition as read-write also, but still cannot edit the interfaces file.

This never happened before, I was always able to add the required routes after interface configuration. 

Does this give any indication or link to the problem at hand? Do i need to rebuild the whole security onion master and sensor VMs?

Regards

Prashant

[Leding Tech]

Prashant,

What are the permissions for the the interfaces file when issuing the ls -l command?

--- Sent from my phone so please excuse typos - Thanks ---

You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/edJiM-ejAmE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.

[Prashant Shrivastva]

Yes regarding the interfaces file, permission was the issue. Got it fixed. A rookie mistake.

But the primary SPAN issue is still not fixed. Not getting the SPAN (mirror) traffic on the server.

Does security onion drops traffic even on tcpdump level? I thought tcpdump, when run against an interface has nothing to do with the underlying application.

Regards

Prashant

[Leding Tech]

Just to make sure I understand your topology...

Your SO box is running as a VM on ESX and has a second interface that receives the traffic from the network you are monitoring...correct?

If so, I have this exact same setup - works fine...

Also, how are you feeding in the monitored traffic to this secondary interface?

--- Sent from my phone so please excuse typos - Thanks ---

[Prashant Shrivastva]

My topology :

security onion master - its a VM, which receives Beat traffic for DNS and AD monitoring

security onion sensor - its a VM on esxi and second  physical interface receives SPAN, (SPAN is configured fine as i receive required traffic when i connect the SPAN cable with wire shark laptop) this sensor is made as forwarder

Now the scenario is that the master was untouched and sensor was reinstalled, giving one virtual interface as SPAN interface and the other for management

When doing tcpdump, the traffic of SPAN is not shown, and neither anything populates on the sguil db. 

From esxi perspective the vswitch was configured to map physical nic of the server with the virtual adapters of the VM. the vswitch config was working fine.

Thanks

Prashant

[Leding Tech]

Ok so this was working fine up until the sensor was reinstalled?

--- Sent from my phone so please excuse typos - Thanks ---

[Prashant Shrivastva]

Yes. Until sensor was reinstalled it was working fine . 

The reason I had to reinstall was because the span cable was pulled out of sensor in a dc activity . But now it's pushed back in . 

What do you suggest ? 

Regards 

Prashant 

Prashant,

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Thanks
with regards,
Prashant Shrivastva
8050674842

--

Thanks
with regards,
Prashant Shrivastva
8050674842

--

Thanks
with regards,
Prashant Shrivastva
8050674842

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/edJiM-ejAmE/unsubscribe.

To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

Thanks
with regards,
Prashant Shrivastva
8050674842

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/edJiM-ejAmE/unsubscribe.

To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--

Thanks
with regards,
Prashant Shrivastva
8050674842

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/edJiM-ejAmE/unsubscribe.

To unsubscribe from this group and all its topics, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Keith Roberson]

I was having sort of the same issue on a rebuild. I ended up deleting my known_hosts from the sensor before a rebuild (sudo -I, the cd into .ssh and rm the known_hosts file). I noticed that if I didn’t do that I would still get a complete setup and I also was able to receive traffic on my forward node using wire shark off the sniffing interface, but nothing was populating on the master node. Good luck. 

Keith Roberson

678-410-8830

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.

[ledin...@gmail.com]

Interesting - do the sensors communicate with the master via SSH?

[ledin...@gmail.com]

Nevermind - just found the answer in the architecture diagram...good call Keith - I bet that is part of or the actual root cause here...

[Keith Roberson]

Great. I hope it helped.

Keith Roberson
678-410-8830

> On Jan 27, 2019, at 22:30, ledin...@gmail.com wrote:
>
> Nevermind - just found the answer in the architecture diagram...good call Keith - I bet that is part of or the actual root cause here...
>

[Prashant Shrivastva]

Hi Keith,

So you mean deleting the known_hosts file  on the sensor, and then run a setup on the sensor?

I did that, but not receiving the SPAN traffic on the interface still.

Now I am making the sensor a standalone security onion server. just to isolate if there are any misconfigs at the master and sensor relation.

So basically i am making the sensor a new standalone deployment, that is input with SPAN. I am doing this to check if everything works fine. This will tell  me if master and sensor relation is at fault or not.

If the standalone works fine, I might need to reinstall master and sensor topology. But if the same issue happens, I am getting my ESXi virtual switch config getting checked.

Shall update you. Thanks.

Regards

Prashant

[Prashant Shrivastva]

Hi,

I reinstalled the sensor VM on esxi, and turns out that the issue of not receiving SPAN traffic still exists. So this might not be a master and sensor sync issue.

I need to review on the ESXi, again. How can i check if the ESXi is receiving SPAN?

I can log into esxi using ssh, can you tell how can I check if the SPAN traffic is hitting esxi? tcpdump doesn't work here.

Thanks

Prashant

[Brian Dorr]

So when you added the nic, did you add it as a pci device? Your NIC should be listed if you set up pass through correctly