Syslog-ng service not working

[haris...@gmail.com]

Hello team,

Our Security Onion sensor stopped sending logs to QRadar. Syslog-ng service is running and we see the Netstat output contains "Close_Wait" for the Syslog service. Restarting the service and rebooting the machine did not solve the issue. This service was working completely fine earlier and we couldn't see any logs from the past 20 days.

Also, we ran tcpdump on listening port 514 on management interface eth0 and we do not see any packets. When compared to a working sensor, we see the sensor listening Syslog data on port 514 on interface 514.

While restarting syslog-ng service, I see these sentences:

Duplicate parser node in radix tree; type='2', name='i1', value='10779'

In the recent days, we have not changed any Syslog-ng configuration. Attached screenshots are few references. Please help us with the situation.

Regards,
Harish

[Wes]

Harish,

Please provide the output of sostat-redacted, attaching as a plain text file, or using a service like Pastebin.com.

Thanks,
Wes

[haris...@gmail.com]

Hello Wes,

I have attached the sostat-redacted output. Please let me know the next steps.

[Wes]

Harish,

It looks like you've not been receiving much traffic after 3/21. Could anything have happened around this time to cause changes to the system?

I also see:

* snort-1 (alert data)[ FAIL ]
* snort-2 (alert data)[ FAIL ]
* snort-3 (alert data)[ FAIL ]
* snort-4 (alert data)[ FAIL ]
* snort-5 (alert data)[ FAIL ]
* snort-6 (alert data)[ FAIL ]
* snort-7 (alert data)[ FAIL ]
* snort-8 (alert data)[ FAIL ]

Have you checked /var/log/nsm/hostname-interface/snortu-x.log for clues?

Also, the format of your sostat looks a little older. When is the last time you ran soup?

Thanks,
Wes

[haris...@gmail.com]

Hello Wes,

We have run soup -y on May 18th and rebooted the machine.
I see the below errors in /var/log/nsm/hostname-interface/snortu-x.log

Unable to connect to x.x.x.x on port 7736.
Trying again in 15 seconds
Listening on port 8101 for barnyard connections.
Error: Invalid snort stats line: ################################### Perfmon stop: pid=5513 at=Fri May 12 23:00:07 2017 (1494630007) ###################################
barnyard connected: sock5 127.0.0.1 44659
Unable to connect to x.x.x.x on port 7736.
Trying again in 15 seconds
Unable to connect to x.x.x.x on port 7736.
Trying again in 15 seconds
Unable to connect to x.x.x.x on port 7736.

Regards,
Harish

[Wes Lambert]

Harish,

What is the output of the following?

snort -V

lsb_release -a

Thanks,
Wes

Regards,
Harish

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.

[haris...@gmail.com]

On Tuesday, May 23, 2017 at 7:23:54 PM UTC+5:30, haris...@gmail.com wrote:

Here is the output:

XXXX@XXXX:/var/log/nsm/XXXXX-eth1$ snort -V

,,_ -*> Snort! <*-
o" )~ Version 2.9.8.0 GRE (Build 229)
'''' By Martin Roesch & The Snort Team: http://www.snort.org/contact#team
Copyright (C) 2014-2015 Cisco and/or its affiliates. All rights reserved.
Copyright (C) 1998-2013 Sourcefire, Inc., et al.
Using libpcap version 1.1.1
Using PCRE version: 8.12 2011-01-15
Using ZLIB version: 1.2.3.4

XXXXX@XXXXX:/var/log/nsm/XXXXX-eth1$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04.5 LTS
Release: 12.04
Codename: precise

FYI. We have other sensors having the same syslog-ng issue where snort-x (alert data) is [OK].

[Wes Lambert]

Harish,

It looks like you are still running the old Security Onion 12.04.  Before trying to troubleshoot any further, I'd recommend you upgrade to Security Onion 14.04.  This will allow you to take advantage of newer software, improved performance, functionality, and bug fixes that may not be present in 12.04.

You can find information about upgrading here:

https://github.com/Security-Onion-Solutions/security-onion/wiki/Security-Onion-14.04-Release-Notes

https://github.com/Security-Onion-Solutions/security-onion/wiki/Upgrading-from-12.04-to-14.04

Thanks,

Wes

[haris...@gmail.com]

Hi Wes,

I don't think we are planning to proceed with Upgrade at this point because there are many sensors with the same version. Current focus is to send all the alerts to Qradar and ensure there is network monitoring from Security perspective. We will plan it in the near future but at this point we would like to troubleshoot this case and resolve it. As I said, there are couple of sensors with the same version having no issues with syslog-ng and are forwarding logs to Qradar. Could you please help us troubleshooting the case ?

Regards,
Harish

[Harish]

Hello Wes & team,

Could you please help us with the below mentioned issue.

Regards,
Harish

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---

You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/gB1Nng5CACo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.

[Wes]

Harish,

I would try first explicitly killing the pid in relation to the syslog-ng CLOSE_WAIT state, (and maybe restarting again (service and/or box)) and see if you get better results.

Thanks,
Wes

[haris...@gmail.com]

On Tuesday, May 23, 2017 at 7:23:54 PM UTC+5:30, haris...@gmail.com wrote:

Hello Wes,

We have attempted this action and failed. Also, I see Snort alert fail on all the sensors now. Is there any action that we can take on the SO Master?

Meanwhile, I shall see firewall connectivity on the ports 7736.

Regards,
Harish

[haris...@gmail.com]

On Tuesday, May 23, 2017 at 7:23:54 PM UTC+5:30, haris...@gmail.com wrote:

Also Wes,

I see this error on few sensors:

Output from /var/log/nsm/sensor/snortu-x.log file:

Reputation config:
WARNING: Can't find any whitelist/blacklist entries. Reputation Preprocessor disabled.

Harish: Not sure of the above warning.

Some more output from the /var/log/nsm/sensor/snortu-x.log file

+++++++++++++++++++++++++++++++++++++++++++++++++++
Initializing rule chains...
ERROR: /etc/nsm/rules/downloaded.rules(1008) Bad protocol: smb.
Fatal Error, Quitting..

Does this mean the sensor couldn't reach master on SMB port(445) ?

Thanks in Advance,
Harish

[Wes]

Harish,

I'm not sure what you mean by attempting and failing. What did you try, and what was the result?

You are likely receiving the WARNING message (notice, that a WARNING should not prevent Snort from working) about the whitelist/blacklist, because you have elected to use the reputation preprocessor, but have no WLs/BLs defined.

The ERROR message you are receiving is likely due to whatever rule (referencing SMB exists at line 1008 of /etc/nsm/rules/downloaded.rules. Try disabling the rule (from the master first, then run rule-update on the master, then sensors), and see if you continue to receive the same error message.

https://github.com/Security-Onion-Solutions/security-onion/wiki/ManagingAlerts#disable-the-sid

Thanks,
Wes

[haris...@gmail.com]

Thanks a ton Wes,

I have followed the same steps(https://github.com/Security-Onion-Solutions/security-onion/wiki/ManagingAlerts#disable-the-sid) and the snort alert service starts working. Also the alerts are being forwarded to QRadar :). The rule - 1008 is with respect to Wannacry signature released last month. Not sure what is wrong with the rule, the snort service fails and is there a way to troubleshoot what is wrong with the signature ?

Signature:
---------
alert smb any any -> $HOME_NET any (msg:"ET EXPLOIT Possible ETERNALBLUE MS17-010 Heap Spray"; flow:to_server,established; content:"|ff|SMB|33 00 00 00 00 18 07 c0 00 00 00 00 00 00 00 00 00 00 00 00 00|"; offset:4; depth:25; content:"|08 ff fe 00 08 41 00 09 00 00 00 10|"; within:12; fast_pattern; content:"|00 00 00 00 00 00 00 10|"; within:8; content:"|00 00 00 10|"; distance:4; within:4; pcre:"/^[a-zA-Z0-9+/]{1000,}/R"; threshold: type threshold, track by_src, count 12, seconds 1; classtype:trojan-activity; sid:2024217; rev:1;)

[Wes Lambert]

Harish,

That signature comes from the ET ruleset, which is a little more optimized for Suricata.

Suricata supports different protocols (including SMB) in the rules:

https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Suricata_Rules#Protocol

Notice the protocols supported by Snort for use in rules:

http://manual-snort-org.s3-website-us-east-1.amazonaws.com/node29.html#SECTION00422000000000000000

You'll find that if you use the ET ruleset(s) with Snort, you may run into some rules that cause it to have issues.

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---

You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.

To post to this group, send email to security-onion@googlegroups.com.

[haris...@gmail.com]

On Thursday, June 1, 2017 at 9:36:07 PM UTC+5:30, Wes wrote:
> Harish,
>
>
> That signature comes from the ET ruleset, which is a little more optimized for Suricata.
>
>
> Suricata supports different protocols (including SMB) in the rules:
>
>
> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Suricata_Rules#Protocol
>
>
>
> Notice the protocols supported by Snort for use in rules:
>
>
> http://manual-snort-org.s3-website-us-east-1.amazonaws.com/node29.html#SECTION00422000000000000000
>
>
> You'll find that if you use the ET ruleset(s) with Snort, you may run into some rules that cause it to have issues.
>
>
> Thanks,
> Wes
>
>
> On Thu, Jun 1, 2017 at 10:55 AM, <haris...@gmail.com> wrote:
> Thanks a ton Wes,
>
>
>
> I have followed the same steps(https://github.com/Security-Onion-Solutions/security-onion/wiki/ManagingAlerts#disable-the-sid) and the snort alert service starts working. Also the alerts are being forwarded to QRadar :).  The rule - 1008 is with respect to Wannacry signature released last month. Not sure what is wrong with the rule, the snort service fails and is there a way to troubleshoot what is wrong with the signature ?
>
>
>
> Signature:
>
> ---------
>
> alert smb any any -> $HOME_NET any (msg:"ET EXPLOIT Possible ETERNALBLUE MS17-010 Heap Spray"; flow:to_server,established; content:"|ff|SMB|33 00 00 00 00 18 07 c0 00 00 00 00 00 00 00 00 00 00 00 00 00|"; offset:4; depth:25; content:"|08 ff fe 00 08 41 00 09 00 00 00 10|"; within:12; fast_pattern; content:"|00 00 00 00 00 00 00 10|"; within:8; content:"|00 00 00 10|"; distance:4; within:4; pcre:"/^[a-zA-Z0-9+/]{1000,}/R"; threshold: type threshold, track by_src, count 12, seconds 1; classtype:trojan-activity; sid:2024217; rev:1;)
>
>
>
>
>
> --
>
> Follow Security Onion on Twitter!
>
> https://twitter.com/securityonion
>
> ---
>
> You received this message because you are subscribed to the Google Groups "security-onion" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
>
> To post to this group, send email to securit...@googlegroups.com.

>
> Visit this group at https://groups.google.com/group/security-onion.
>
> For more options, visit https://groups.google.com/d/optout.

Hello Wes,

We are seeing this issue again. We see rule-update successfully working. Syslog services are good. But not sure the snort alerts not being forwarded to QRadar. could you please help us on the same ?

[haris...@gmail.com]

Hello Wes & team,

Please look into and update us. Thanks in Advance.

Harish

[Wes]

Harish,

Have you gone through all of the steps above? Have you checked the nsort log for clues?

Thanks,
Wes

[haris...@gmail.com]

Yes Wes,

I dont see any errors in the log file - /var/log/nsm/sensor/snortu-x.log file. I have compared it with a working sensor and I dont see any issues or errors in the snorty-x.log files.

Any other checks ? Do you want me to pull out any conf files or log files for you to refer ?

Regards,
Harish

[Wes]

Harish,

Please provide the output of sostat-redacted for the problem machine.

Thanks,
Wes

[haris...@gmail.com]

Hello Wes,

I have attached sostat-redacted output. Thanks in advance.

Regards,
Harish

[haris...@gmail.com]

Hello Wes,

I see this error in the packet sent to QRadar. Can you please help me if something is wrong?

root@XXXXXXXXX:/etc/nsm/ubuntu-eth4# tcpdump -n dst port 514 -v
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
13:44:29.596319 IP (tos 0x0, ttl 64, id 7229, offset 0, flags [DF], proto TCP (6), length 52)
X.X.X.X.53265 > X.X.X.X.514: Flags [F.], cksum 0x0de3 (incorrect -> 0x13db), seq 3785828257, ack 500806630, win 1024, options [nop,nop,TS val 584124928 ecr 2998977654], length 0

Regards,
Harish

[Wes Lambert]

Harish,

It does not appear rule-update is working as it should:

=========================================================================
IDS Rules Update
=========================================================================
Sun Jul 16 07:01:01 UTC 2017
Backing up current local_rules.xml file.
Cleaning up local_rules.xml backup files older than 30 days.
Backing up current downloaded.rules file before it gets overwritten.
Cleaning up downloaded.rules backup files older than 30 days.
Backing up current local.rules file before it gets overwritten.
Cleaning up local.rules backup files older than 30 days.
Sleeping for 60 minutes to allow master time to download new rules.
ssh: connect to host X.X.X.X port 22: Connection timed out
Copying rules from X.X.X.X.
ssh: connect to host X.X.X.X port 22: Connection timed out
ssh: connect to host X.X.X.X port 22: Connection timed out
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,password).
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,password).
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,password).
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,password).
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,password).

It also appears as though your sensor's API key could not be found on the master.

I would really suggest upgrading to Security Onion 14.04 as mentioned before, for better stability, bug fixes, etc.

Thanks,

Wes

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

[Harish]

Thanks Wes,

We shall consider your suggestion. For now, is there a way to generate a new API key ?

I shall look into the rule-update, we see some issues while adding the public and private keys.

Regards,
Harish

You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/gB1Nng5CACo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.

[Wes]

> Permission denied (publickey,password).It also appears as though your sensor's API key could not be found on the master.I would really suggest upgrading to Security Onion 14.04 as mentioned before, for better stability, bug fixes, etc.Thanks,Wes

> --
>
> Follow Security Onion on Twitter!
>
> https://twitter.com/securityonion
>
> ---
>
> You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
>
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/gB1Nng5CACo/unsubscribe.
>
> To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
>
> To post to this group, send email to securit...@googlegroups.com.
>
> Visit this group at https://groups.google.com/group/security-onion.
>
> For more options, visit https://groups.google.com/d/optout.

Harish,

For the APIKEY, try having a look here:

https://groups.google.com/d/msg/security-onion/CgcZNzv-h-0/7V2yHW4lBgAJ

Thanks,
Wes

[haris...@gmail.com]

Hello Wes,

Though I tried regsitering API key I did not see valid API check on the sostat output. However I see connection established between the SO sensor & QRadar but when I run tcpdump I see below packet. Can you please help me if you understand this one.

root@<sensor>:/etc/nsm/sensor-eth4# tcpdump -vvn dst <QRadar IP>
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
13:58:15.146519 IP (tos 0x0, ttl 64, id 3653, offset 0, flags [DF], proto TCP (6), length 52)
<Sensor IP>.35770 > <QRadar IP>.514: Flags [.], cksum 0xe553 (incorrect -> 0xa94c), seq 3798946483, ack 1280434003, win 1024, options [nop,nop,TS val 134073678 ecr 1188288927], length 0

Not sure why it shows incorrect in the packet checksum. No clue about it.

Regards,
Harish