Sguil unable to connect on port 7734

[Robert Campbell]

Hello everyone,
I started experiencing an issue this morning that has left me bewildered.

We have an SO server and distributed SO clients in various locations.
The server and clients were rebuilt at the beginning of this week.
We connect to the server via an SO VM with the following shell syntax to forward traffic:
sudo ssh -X user@soserver -L443:localhost:443 -L444:localhost:444 -L7734:localhost:7734 -L3154:localhost:3154 -L9876:localhost:9876

This configuration has been working and was working yesterday evening. This morning we are constantly getting "Unable to connect to SERVER on port 7734" error messages.

In the past the issue has either been not waiting long enough after a reboot or a large number of uncategorized alerts. Neither of those are the issue today.

I have attached relevant information from my troubleshooting efforts:

The redacted sostat is attached.

The following errors are produced when running sostat:

ls: cannot access /nsm/sensor_data/*/dailylogs/: No such file or directory
/usr/bin/sostat: line 82: cd: /nsm/sensor_data/*/dailylogs: No such file or directory
/usr/bin/sostat: line 84: cd: OLDPWD not set
grep: /nsm/sensor_data/*/snort-*.stats: No such file or directory

I have ruled out a large number of uncategorized events as being the problem:

Sguil Uncategorized Events
=========================================================================
COUNT(*)
836

nsm --all --status output:
Status: SOserver
* sguil server [ OK ]
Status: HIDS
* ossec_agent (sguil) [ OK ]

/var/log/nsm/securityonion/sguild.log attached (sanitized)

netstat -na | grep -i listen (attached netstat_listen.txt)

Hoping this is something simple I overlooked that a fresh pair of eyes can quickly uncover.

Thank you in advance for your time and help,

Robert

[Matt .]

I don't claim to be an expert by any means. So please take my thoughts with a grain of salt. I could be mistaken on the implications of my observations! :)

I don't see a 2nd NIC listed, is there one usually? Is it "down"?

I don't see status for the rest of the processes, like netsniff-ng, pcap_agent, snort etc (should be right after HIDS status). Have you checked other logs for those agents?

I see 28,000 enabled rules. Normally it's recommended to keep it down to 5,000-6,000 or so.

Good luck, and I'm sure someone else more knowledgeable will reply before long.

[Robert Campbell]

Matt,
Thank you for the quick reply.

This is a server only and is not acting as an agent. This is why it only has a single NIC. This also explains the missing processes normally seen after the HIDS status. I know it seems odd.

We just switched from VRT to ETPro, so another round of pruning must begin to get the number of rules down to a reasonable level ;-)

I appreciate the time to look and help me look at this!

[Heine Lysemose]

Hi

It seems like nothing is listening on port 7734.
Have you tried restarting the Sguil server?

Regards,
Lysemose

--
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at http://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[Robert Campbell]

It seems like nothing is listening on port 7734.
Have you tried restarting the Sguil server?

I was unable to see anything listening on this port either before or after rebooting the server this morning. It seems to me that it should be listening and if not, there should be an error somewhere that I can identify...

Restarted NSM and netstat still shows no listening on 7734; grep for 7734 shows nothing

Output attached.

[Robert Campbell]

That is exactly how it looks

[Robert Campbell]

Gary,
I looked at it and included a redacted copy of it in the initial post. Maybe you can find clues in there I missed!

[Doug Burks]

From your sguild.log:

2015-08-26 16:04:01 pid(6795) UPDATE `event_SOsensor-eth1_20150824`
SET status=1, last_modified='2015-08-26 16:04:01', last_uid='1' WHERE
sid=15 AND cid=1136
mysqlexec/db server: Table
'securityonion_db.event_SOsensor-eth1_20150824' doesn't exist

My guess is you have some database table corruption. Did you have a
power outage or other ungraceful shutdown?

I'd start with running "sudo sguil-db-purge" and see if it's able to
fix the table corruption automatically. If that doesn't help, you can
try standard mysql tools.

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.

--
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com

[Robert Campbell]

Thank you much Doug.

It was a mysql issue after all.

"sudo sguil-db-purge" did not resolve the issue.

After fumbling with this for a while, I used the following:

mysql> UPDATE event SET status=1, last_modified='2015-08-24 00:00:01', last_uid='sguil' WHERE event.status=0 and event.signature LIKE '%';

I restarted the server and all is working well now with sguil.

It appears that a nightly back-up caused the corruption.

Thank you!