Reoccurring install issue

[cm0s...@gmail.com]

I have installed and wiped my drive 5 times now. I have an issue that I can't stop from occurring. I install from scratch to a clean drive, I boot and use Byobu to run 'soup' and upgrade the system. I reboot. I go into the setup and run and configure my 2 network cards then reboot. I then open setup again and choose (I have tried evaluation mode and stand alone modes) and tried Snort and Suricata both. I have tried the rules that don't need an Oinkcode and ones that DO need the code. The only way I can get the system to halfway work is NOT TO UPDATE. It kinda works IF I don't update. Something is breaking it but I am not smart enough to figure it out. I have a sostat-redacted attached. Any hints at what is causing the issue and what I can do to end it would be greatly appreciated.

[Wes]

On Saturday, August 13, 2016 at 12:03:56 PM UTC-4, cm0s...@gmail.com wrote:
> I have installed and wiped my drive 5 times now. I have an issue that I can't stop from occurring. I install from scratch to a clean drive, I boot and use Byobu to run 'soup' and upgrade the system. I reboot. I go into the setup and run and configure my 2 network cards then reboot. I then open setup again and choose (I have tried evaluation mode and stand alone modes) and tried Snort and Suricata both. I have tried the rules that don't need an Oinkcode and ones that DO need the code. The only way I can get the system to halfway work is NOT TO UPDATE. It kinda works IF I don't update. Something is breaking it but I am not smart enough to figure it out. I have a sostat-redacted attached. Any hints at what is causing the issue and what I can do to end it would be greatly appreciated.

What do you mean, by it not working? What specific issues are you experiencing?

From your sostat, it looks like Suricata is failing to run as it should, and it looks like it may be related to PF_RING:

grep: /proc/net/pf_ring/*-*: No such file or directory
grep: /proc/net/pf_ring/*-*: No such file or directory
grep: /proc/net/pf_ring/*-*: No such file or directory

I also only see one interface (eth0). If this is a standalone, I would expect to see two interfaces.

What is the output of the following?

sudo more /etc/nsm/sensortab

Is the second interface (the one to be used for sniffing) configured?

You may want to try re-running setup to configure an appropriate sniffing interface.

If you are still having issues after doing so, please provide output of the following:

uname -a

sudo apt-get install --reinstall securityonion-pfring-module

sudo soup

Thanks,
Wes

[cm0s...@gmail.com]

Ok. I fresh installed and ran for 15 days and all went very well. Very pleased. I ran "sudo soup" today and did a complete update. My system now doesn't work again. When I run sostat-quick there are no rules and I did the rule-update and the reinstalled security onion-pfring-module. I still have 0 rules. Only way it works on this machine is to NOT update. I am also getting a -warning on the software updater telling me the HWE reached end-of-life on 2016-08-04. SOUP doesn't address this issue. I have not applied the HWE patch from ubuntu. I did include the info you requested in a txt file.

[Wes]

So we can get a little bit more information about your current setup, could you please attach the output of sostat-redacted?

Thanks,
Wes

[cm0s...@gmail.com]

[cm0s...@gmail.com]

Yes....here is the redacted.

[Wes Lambert]

I'm trying to understand what your current issue is...if you didn't have rules, you wouldn't have SO alerting on anything, and you wouldn't see stuff similar to the following (from your sostat):

=========================================================================

Sguil Uncategorized Events

=========================================================================

COUNT(*)

494

=========================================================================

Sguil events summary for yesterday

=========================================================================

Totals GenID:SigID Signature

11 1:2522196 ET TOR Known Tor Relay/Router (Not Exit) Node Traffic group 99

Total

11

=========================================================================

Top 50 All time Sguil Events

=========================================================================

Totals GenID:SigID Signature

63 1:2523332 ET TOR Known Tor Relay/Router (Not Exit) Node Traffic group 667

50 1:2523328 ET TOR Known Tor Relay/Router (Not Exit) Node Traffic group 665

23 1:2522396 ET TOR Known Tor Relay/Router (Not Exit) Node Traffic group 199

20 1:2522196 ET TOR Known Tor Relay/Router (Not Exit) Node Traffic group 99

What is the output of "sudo rule-update"?

In regard to the HWE stack:

The 14.04.5.1 ISO was just released for testing and will come shipped with the new Ubuntu 16.04 Xenial HWE stack.  

https://github.com/Security-Onion-Solutions/security-onion/issues/879

This link should walk you through how to update this on an existing installation

https://wiki.ubuntu.com/Kernel/LTSEnablementStack

Ex. sudo apt-get install --install-recommends linux-generic-lts-xenial 

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[cm0s...@gmail.com]

> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
>
> To post to this group, send email to securit...@googlegroups.com.

>
> Visit this group at https://groups.google.com/group/security-onion.
>
> For more options, visit https://groups.google.com/d/optout.

Thanks for link. My current issue is that I DON'T have rules now. All those things you copy/pasted in previous post (sguil) were from the previous 15 days of running where everything worked great. My SUDO SOUP did not eliminate my logs or change ANYTHING except I no longer list any rules in sostat. Where before it listed like 12000 rules it now lists 0. My squil is no longer listing NEW events and I'm sure they didn't just stop. This same issue reoccurs every time I do a sudo soup after 15-30 days of running. It breaks it. No rules. Every time.

[Wes]

Thanks,
Wes

[cm0s...@gmail.com]

Ok...I did the rule update. Previously I did the module update. I have included the output from the rule update and I did another sostat-quick and copied a couple pages. All are included in the attached file.

[Wes]

It looks like sostat-quick is not picking up the enabled rules. It also looks like you currently have over 27k enabled (from the rule-update output).

Try the following:

sudo sostat | grep -m 1 "Enabled Rules:"

Then try:

sudo

I think what is happening is that after you are running soup, the pulledpork.log has been altered, so that /usr/bin/sostat cannot retrieve the last 1000 lines. Therefore, when you run /usr/bin/sostat-quick, all it is doing is grepping the output of sostat, which of course, has no output for PulledPork. You can check /var/log/nsm/pulledpork.log to see each time the rules have been updated by PulledPork.

I'm not seeing any immediate issue with any of the services from the sostat output. Are you still experiencing issues with Sguil?

If so, have you tried checking /var/log/nsm/securityonion/sguild.log or the Snort or barnyard logs? Keep in mind, if you are not F8'ing alerts from the console, they will only increment the alert count and not show as "new" alerts.

https://github.com/Security-Onion-Solutions/security-onion/wiki/Help

You can also test the rules/Snort/Sguil by using tcpreplay to replay some PCAPS:

Ex. sudo tcpreplay -ieth1 -M10 /opt/samples/*

Thanks,
Wes

[Wes]

Forget about:

"Then try:

sudo"

[cm0s...@gmail.com]

0.0 0.0 grep --color=auto -m 1 Enabled Rules:

sudo tcpreplay -ieth1 -M10 /opt/samples/*

processing file: /opt/samples/6to4.pcap
processing file: /opt/samples/best_malware_protection.pcap
Warning in tcpreplay.c:replay_file() line 279:
/opt/samples/best_malware_protection.pcap DLT (RAW) does not match that of the outbound interface: eth1 (EN10MB)
processing file: /opt/samples/bredolab-sample.pcap
processing file: /opt/samples/ConfickerB9hrs.pcap
Warning in send_packets.c:send_packets() line 178:
Unable to send packet:
Warning in send_packets.c:send_packets() line 178:
Unable to send packet:
Warning in send_packets.c:send_packets() line 178:
Unable to send packet:
Warning in send_packets.c:send_packets() line 178:
Warning in tcpreplay.c:replay_file() line 279:
/opt/samples/emerging-all.pcap DLT (RAW) does not match that of the outbound interface: eth1 (EN10MB)
Warning in send_packets.c:send_packets() line 178:
Unable to send packet: Error with PF_PACKET send() [17872]: Message too long (errno = 90)
Warning in send_packets.c:send_packets() line 178:
Unable to send packet: Error with PF_PACKET send() [18301]: Message too long (errno = 90)

[Wes]

I'm not sure why sostat is not picking up the lines from pulledpork.log. You should be able tail pulledpork.log yourself (as sostat normally does) by doing the following:

sudo tail -1000 /var/log/nsm/pulledpork.log

The tcpreplay command was more so to see if any alerts were generated by Snort, populated into Sguil, etc.

Do you see the events populating in Sguil?

Thanks,
Wes

[cm0s...@gmail.com]

Here is a copy of the tail pulledpork. I have seen no new events since the upgrade though I know events are happening daily. I watched them populate every day for 15 days. It was excellent. I have had not a single entry since 'sudo soup'. Should I go back in and try to set up everything again? I wouldn't think that would be necessary. In times past when I tried to run setup again it just cause more issues than I was originally having. Something is changing dramatically when I run the sudo soup. I can't figure out what it is. Anyway, here is a copy of todays pulledpork.

[Wes]

As far as the rule updates, it looks like you are getting a 500 error during the daily rule-update. Are you running behind a proxy? If so, you may want to view the following:

https://github.com/Security-Onion-Solutions/security-onion/wiki/Proxy

In regard to not being able to see the Sguil alerts, try setting "DEBUG 2" in /etc/sguil/sguil.conf and tailing sguild.log in a separate terminal window (sudo tail -f /var/log/nsm/securityonion/sguild.log) while you use tcpreplay to replay some PCAPs.

If you do not see anything there, try seeing if the alerts are picked up by snort_agent by taking a look at /var/log/nsm/[hostname-interface]/snort_agent.log.

If you do not see anything there, try taking a look at /var/log/nsm/[hostname-interface]/barnyard.log to see if there are any issues there.

You may also want to check /var/log/nsm/[hostname-interface]/suricata.log and running the following on securityonion_db just in case:

sudo mysqlcheck -c securityonion_db


Thanks,
Wes

[cm0s...@gmail.com]

Ok. Nothing you had listed worked. I decided to try the setup again even though it has never worked for me in the past. I guess I did something right. Maybe it was adding the export no_proxy="localhost,127.0.0.1" to /etc/environment. When I ran the setup this time it worked. My system is now back online 100% with the updates. Very happy. Case is now closed........for now. I included my Page 6 enabled rules in a text file to show how it normally reads for me. Thanks for the help. Something you had me to do worked this time.

[cm0s...@gmail.com]

New issue. Netsnif-ng is now maxing out one of my processors. It didn't before but is now. Even when I shut down my eth1 it still maxes it out. The number of rules have not changed. Is there a closed loop happening? Since I am able to capture now I hate killing the netsnif. Any suggestions??

[Wes]

Please post this issue to a new thread--netsniff-ng is assigned a single core to itself by default with SO 14.04. Could it be that you are experiencing a traffic spike, or had not previously noticed the load due to the other issues you were having? What kind of (amount of) traffic are you currently monitoring?

Thanks,
Wes

[cm0s...@gmail.com]

No. I spend more hours trying to make it work again and failing than is allowable. For my system there is no 'sudo soup' to ever be run. Fresh install with no updates and it works 100% as it should. I don't understand why the updating ruins the system but it does. And it just gets worse the more I try to "fix" it. The system is a quad core intel with 8 gig memory and 2 1ghz nic cards. The drives are plenty large enough. It is just something happens to the operating system during an update that changes too many things and I would need to be a software engineer to track them all down. I have found several issues it causes and fixed a few but there are just too many. Maybe your next release will be more stable with the updating.