VRT Rules not in IDS

[Lonejeeper]

All,

We've got a subscription to the snort VRT ruleset, but it appears that the VRT rules aren't making their way into the IDS. I've been working with the VRT team a bit, but I'm tempted to believe that the issue is more likely to be on my end.

I've verified that the rule_url is correctly formatted. When I run rule-update I see that the rule counts change, so it's getting something done. When I search the downloaded.rules and sid-msg.map for entries specific to the VRT rules I come up with nothing. We also subscribe to the Emerging Threats rules, and they appear to be incorporated.

I've tried to only have the VRT rule_url entry and it has made no difference.

I'm not 100% sure that my means of validation are correct... I assume you can reasonably expect the newly downloaded rules to be in both the downloaded.rules and sid-msg.map, right? Is there something else worth checking?

Thanks!

[Jeremy Hoel]

VRT rules are not prefaced with VRT. They show up as normal rules.

What do you think you are missing? If you look at your rules file and
'grep -v "ET\ " <rules file>' are there a number of rules left? Do
you have the community ruleset too or just VRT and ET?

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/groups/opt_out.

[Lonejeeper]

Part of the problem is that I don't know that my verification method is effective.

I've gone to the snort vrt advisories page (https://www.snort.org/vrt/docs/ruleset_changelogs/2953/changes-2013-10-17.html ) and grepped through the downloaded.rules and sid-msg.map files for a few randomly chosen key words, like virus names. The rules are in the manually downloaded tar.gz file, but don't appear to be in my sensor.

I also have an ETPRO subscription, but that appears to have been incorporated.

[Doug Burks]

Did you try Jeremy's suggestion?

grep -v "ET\ " /etc/nsm/rules/downloaded.rules

Do you have a separate server and sensor? Run that command on both
and compare the output.

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/groups/opt_out.

--
Doug Burks
http://securityonion.blogspot.com

[Lonejeeper]

On Tuesday, October 22, 2013 8:20:15 AM UTC-4, Doug Burks wrote:
> Did you try Jeremy's suggestion?
>

I sometimes worry that I'm mistaken for being more competent at this sort of thing than I really am.

It may be that I'm just confused... Grepping "ET" would get me the emerging threats rules, right? I've already verified that the emerging threats "pro" rules are being incorporated, it's the subscription-based sourcefire/snort rules (that I'm calling VRT) I'm trying to verify.

Anyway, yes, when I run "grep -v "ET\ " /etc/nsm/rules/downloaded.rules " I do get output.

I'm looking for "1:28244 <-> ENABLED <-> MALWARE-CNC WIN.Trojan.Phrovon outbound connection attempt (malware-cnc.rules)" specifically, as it was added on 10/17/2013.

When I run "cat downloaded.rules | grep "Phrovon" I get no results. When I grep downloaded.rules for "Trojan" I get a lot of results. I don't have any dropsid.conf or disablesid.conf that would disable/remove rules of that type.

My oinkcode does allow me to download the pay-for rules when I do so manually... in those downloaded rules I can find the Phrovon entry.

[Doug Burks]

Replies inline.

On Tue, Oct 22, 2013 at 8:54 AM, Lonejeeper <rockc...@gmail.com> wrote:
> On Tuesday, October 22, 2013 8:20:15 AM UTC-4, Doug Burks wrote:
>> Did you try Jeremy's suggestion?
>>
>
> I sometimes worry that I'm mistaken for being more competent at this sort of thing than I really am.
>
> It may be that I'm just confused... Grepping "ET" would get me the emerging threats rules, right? I've already verified that the emerging threats "pro" rules are being incorporated, it's the subscription-based sourcefire/snort rules (that I'm calling VRT) I'm trying to verify.

"grep -v" will only show lines that DO NOT MATCH the desired criteria.
In this case, we're asking grep to show only the rules that do not
contain "ET ".

> Anyway, yes, when I run "grep -v "ET\ " /etc/nsm/rules/downloaded.rules " I do get output.

How much output? How many non-ET rules do you have?

grep -v "ET\ " /etc/nsm/rules/downloaded.rules | wc -l

> I'm looking for "1:28244 <-> ENABLED <-> MALWARE-CNC WIN.Trojan.Phrovon outbound connection attempt (malware-cnc.rules)" specifically, as it was added on 10/17/2013.

Are you a paid VRT subscriber? If so, did you add your paid VRT
oinkcode to /etc/nsm/pulledpork/pulledpork.conf? If not, you won't
have access to rules released in the last 30 days.

> When I run "cat downloaded.rules | grep "Phrovon" I get no results. When I grep downloaded.rules for "Trojan" I get a lot of results. I don't have any dropsid.conf or disablesid.conf that would disable/remove rules of that type.
>
> My oinkcode does allow me to download the pay-for rules when I do so manually... in those downloaded rules I can find the Phrovon entry.
>

[Lonejeeper]

Ah, ok. Thanks for that explanation... I was missing the point there.

grep -v -c "ET\ " /etc/nsm/rules/downloaded.rules yields 625.

Yes, we are a paid VRT subscriber. I've just verified it is still active. I only see a single oinkcode, I believe it was upgraded to subscriber status during the subscription process. We are also a paid ETPRO subscriber. I asked for one or the other and got both. The ETPRO rules do appear to be integrated.

Yes, my oinkcode is in pulledpork.conf.

sample:

rule_url=https://www.snort.org/sub-rules/|snortrules-snapshot.tar.gz|myoinkcode

rule_url=https://rules.emergingthreatspro.com/|etpro.rules.tar.gz|myETPROcode

it's my understanding that multple rule_url entries are acceptable. I tried having a single entry regardless and found that the downloaded.rules still did not include the entries I expected to find.

[Doug Burks]

Yes, multiple rule_url entries should work. However, you probably
don't want to enable all ETPRO rules and all VRT rules at the same
time. You should pick and choose the rules that you really need to
protect your environment.

Let's try simplifying things. Please comment out the ETPRO rule_url
and then run the following:
sudo /usr/bin/pulledpork.pl -c /etc/nsm/pulledpork/pulledpork.conf -vvv

Paste all output into your reply (redacting sensitive info as necessary).

[Lonejeeper]

That appears to have shaken something loose... but I don't know why.

results of sudo /usr/bin/pulledpork.pl -c /etc/nsm/pulledpork/pulledpork.conf -vvv

http://code.google.com/p/pulledpork/
_____ ____
`----,\ )
`--==\\ / PulledPork v0.6.1 the Smoking Pig <////~
`--==\\/
.-~~~~-.Y|\\_ Copyright (C) 2009-2011 JJ Cummings
@_/ / 66\_ cumm...@gmail.com
| \ \ _(")
\ /-| ||'--' Rules give me wings!
\_\ \_\\
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Config File Variable Debug /etc/nsm/pulledpork/pulledpork.conf
snort_path = /usr/bin/snort
enablesid = /etc/nsm/pulledpork/enablesid.conf
modifysid = /etc/nsm/pulledpork/modifysid.conf
rule_path = /etc/nsm/rules/downloaded.rules
ignore = deleted.rules,experimental.rules
rule_url = ARRAY(0x2cbf018)
sid_changelog = /var/log/sid_changes.log
sid_msg = /etc/nsm/rules/sid-msg.map
config_path = /etc/nsm/templates/snort/snort.conf
sostub_path = /etc/nsm/rules/so_rules.rules
temp_path = /tmp
distro = Ubuntu-12-04
version = 0.6.0
sorule_path = /usr/local/lib/snort_dynamicrules/
disablesid = /etc/nsm/pulledpork/disablesid.conf
dropsid = /etc/nsm/pulledpork/dropsid.conf
local_rules = /etc/nsm/rules/local.rules,/etc/nsm/rules/decoder-events.rules,/etc/nsm/rules/stream-events.rules,/etc/nsm/rules/http-events.rules,/etc/nsm/rules/smtp-events.rules
MISC (CLI and Autovar) Variable Debug:
arch Def is: x86-64
Config Path is: /etc/nsm/pulledpork/pulledpork.conf
Distro Def is: Ubuntu-12-04
Disabled policy specified
local.rules path is: /etc/nsm/rules/local.rules,/etc/nsm/rules/decoder-events.rules,/etc/nsm/rules/stream-events.rules,/etc/nsm/rules/http-events.rules,/etc/nsm/rules/smtp-events.rules
Rules file is: /etc/nsm/rules/downloaded.rules
Path to disablesid file: /etc/nsm/pulledpork/disablesid.conf
Path to dropsid file: /etc/nsm/pulledpork/dropsid.conf
Path to enablesid file: /etc/nsm/pulledpork/enablesid.conf
Path to modifysid file: /etc/nsm/pulledpork/modifysid.conf
sid changes will be logged to: /var/log/sid_changes.log
sid-msg.map Output Path is: /etc/nsm/rules/sid-msg.map
Snort Version is: 2.9.5.3
Snort Config File: /etc/nsm/templates/snort/snort.conf
Snort Path is: /usr/bin/snort
SO Output Path is: /usr/local/lib/snort_dynamicrules/
SO Stub File is: /etc/nsm/rules/so_rules.rules
Verbose Flag is Set
Base URL is: https://www.snort.org/sub-rules/|snortrules-snapshot.tar.gz|MyOinkCode
Checking latest MD5 for snortrules-snapshot-2953.tar.gz....
Fetching md5sum for: snortrules-snapshot-2953.tar.gz.md5
most recent rules file digest: c2df705033294ea0ce4d5197a42d95f9
current local rules file digest: 1a93f959476c2f48e59c9930a5bd150b
The MD5 for snortrules-snapshot-2953.tar.gz did not match the latest digest... so I am gonna fetch the latest rules file!
Rules tarball download of snortrules-snapshot-2953.tar.gz....
Fetching rules file: snortrules-snapshot-2953.tar.gz
storing file at: /tmp/snortrules-snapshot-2953.tar.gz

current local rules file digest: c2df705033294ea0ce4d5197a42d95f9
The MD5 for snortrules-snapshot-2953.tar.gz matched c2df705033294ea0ce4d5197a42d95f9
so I'm not gonna download the rules file again suckas!
Prepping rules from snortrules-snapshot-2953.tar.gz for work....
extracting contents of /tmp/snortrules-snapshot-2953.tar.gz...
Ignoring plaintext rules: deleted.rules
Ignoring plaintext rules: experimental.rules
Extracted: /tha_rules/VRT-server-other.rules
Extracted: /tha_rules/VRT-pua-adware.rules
Extracted: /tha_rules/VRT-misc.rules
Extracted: /tha_rules/VRT-malware-backdoor.rules
Extracted: /tha_rules/VRT-indicator-compromise.rules
Extracted: /tha_rules/VRT-file-pdf.rules
Extracted: /tha_rules/VRT-content-replace.rules
Extracted: /tha_rules/VRT-file-identify.rules
Extracted: /tha_rules/VRT-browser-webkit.rules
Extracted: /tha_rules/VRT-protocol-telnet.rules
Extracted: /tha_rules/VRT-file-office.rules
Extracted: /tha_rules/VRT-specific-threats.rules
Extracted: /usr/local/lib/snort_dynamicrules/dos.so
Extracted: /tha_rules/VRT-protocol-tftp.rules
Extracted: /tha_rules/VRT-file-java.rules
Extracted: /tha_rules/VRT-local.rules
Extracted: /tha_rules/VRT-rpc.rules
Extracted: /tha_rules/VRT-dns.rules
Extracted: /tha_rules/VRT-protocol-dns.rules
Extracted: /tha_rules/VRT-os-other.rules
Extracted: /tha_rules/VRT-snmp.rules
Extracted: /tha_rules/VRT-protocol-scada.rules
Extracted: /tha_rules/VRT-policy-other.rules
Extracted: /tha_rules/VRT-web-coldfusion.rules
Extracted: /tha_rules/VRT-protocol-voip.rules
Extracted: /tha_rules/VRT-file-image.rules
Extracted: /tha_rules/VRT-chat.rules
Extracted: /tha_rules/VRT-voip.rules
Extracted: /tha_rules/VRT-os-solaris.rules
Extracted: /tha_rules/VRT-server-mssql.rules
Extracted: /tha_rules/VRT-pop3.rules
Extracted: /tha_rules/VRT-os-mobile.rules
Extracted: /tha_rules/VRT-preprocessor.rules
Extracted: /tha_rules/VRT-policy-social.rules
Extracted: /tha_rules/VRT-protocol-ftp.rules
Extracted: /tha_rules/VRT-server-webapp.rules
Extracted: /tha_rules/VRT-protocol-rpc.rules
Extracted: /tha_rules/VRT-server-oracle.rules
Extracted: /usr/local/lib/snort_dynamicrules/misc.so
Extracted: /tha_rules/VRT-server-samba.rules
Extracted: /tha_rules/VRT-scada.rules
Extracted: /tha_rules/VRT-other-ids.rules
Extracted: /tha_rules/VRT-server-apache.rules
Extracted: /tha_rules/VRT-sql.rules
Extracted: /tha_rules/VRT-protocol-nntp.rules
Extracted: /tha_rules/VRT-icmp.rules
Extracted: /tha_rules/VRT-indicator-scan.rules
Extracted: /tha_rules/VRT-file-multimedia.rules
Extracted: /tha_rules/VRT-pua-p2p.rules
Extracted: /tha_rules/VRT-info.rules
Extracted: /tha_rules/VRT-pua-other.rules
Extracted: /tha_rules/VRT-protocol-snmp.rules
Extracted: /tha_rules/VRT-server-mail.rules
Extracted: /tha_rules/VRT-netbios.rules
Extracted: /usr/local/lib/snort_dynamicrules/web-iis.so
Extracted: /tha_rules/VRT-smtp.rules
Extracted: /tha_rules/VRT-protocol-icmp.rules
Extracted: /tha_rules/VRT-sensitive-data.rules
Extracted: /usr/local/lib/snort_dynamicrules/bad-traffic.so
Extracted: /tha_rules/VRT-indicator-shellcode.rules
Extracted: /tha_rules/VRT-web-iis.rules
Extracted: /tha_rules/VRT-protocol-finger.rules
Extracted: /tha_rules/VRT-botnet-cnc.rules
Extracted: /tha_rules/VRT-pua-toolbars.rules
Extracted: /tha_rules/VRT-mysql.rules
Extracted: /tha_rules/VRT-virus.rules
Extracted: /tha_rules/VRT-protocol-imap.rules
Extracted: /tha_rules/VRT-malware-cnc.rules
Extracted: /tha_rules/VRT-web-misc.rules
Extracted: /tha_rules/VRT-tftp.rules
Extracted: /usr/local/lib/snort_dynamicrules/imap.so
Extracted: /tha_rules/VRT-shellcode.rules
Extracted: /tha_rules/VRT-blacklist.rules
Extracted: /tha_rules/VRT-spyware-put.rules
Extracted: /tha_rules/VRT-exploit.rules
Extracted: /tha_rules/VRT-protocol-services.rules
Extracted: /tha_rules/VRT-browser-ie.rules
Extracted: /tha_rules/VRT-ddos.rules
Extracted: /tha_rules/VRT-os-windows.rules
Extracted: /usr/local/lib/snort_dynamicrules/multimedia.so
Extracted: /usr/local/lib/snort_dynamicrules/smtp.so
Extracted: /tha_rules/VRT-attack-responses.rules
Extracted: /usr/local/lib/snort_dynamicrules/snmp.so
Extracted: /tha_rules/VRT-browser-firefox.rules
Extracted: /tha_rules/VRT-browser-chrome.rules
Extracted: /usr/local/lib/snort_dynamicrules/web-client.so
Extracted: /tha_rules/VRT-telnet.rules
Extracted: /tha_rules/VRT-browser-other.rules
Extracted: /tha_rules/VRT-icmp-info.rules
Extracted: /tha_rules/VRT-os-linux.rules
Extracted: /tha_rules/VRT-indicator-obfuscation.rules
Extracted: /tha_rules/VRT-policy-spam.rules
Extracted: /tha_rules/VRT-malware-tools.rules
Extracted: /tha_rules/VRT-x11.rules
Extracted: /usr/local/lib/snort_dynamicrules/chat.so
Extracted: /tha_rules/VRT-p2p.rules
Extracted: /tha_rules/VRT-scan.rules
Extracted: /tha_rules/VRT-ftp.rules
Extracted: /usr/local/lib/snort_dynamicrules/nntp.so
Extracted: /tha_rules/VRT-malware-other.rules
Extracted: /tha_rules/VRT-web-php.rules
Extracted: /tha_rules/VRT-web-activex.rules
Extracted: /tha_rules/VRT-decoder.rules
Extracted: /tha_rules/VRT-rservices.rules
Extracted: /tha_rules/VRT-web-frontpage.rules
Extracted: /tha_rules/VRT-file-executable.rules
Extracted: /usr/local/lib/snort_dynamicrules/netbios.so
Extracted: /tha_rules/VRT-file-other.rules
Extracted: /tha_rules/VRT-backdoor.rules
Extracted: /usr/local/lib/snort_dynamicrules/exploit.so
Extracted: /usr/local/lib/snort_dynamicrules/web-misc.so
Extracted: /tha_rules/VRT-multimedia.rules
Extracted: /tha_rules/VRT-web-client.rules
Extracted: /tha_rules/VRT-exploit-kit.rules
Extracted: /tha_rules/VRT-protocol-pop.rules
Extracted: /tha_rules/VRT-browser-plugins.rules
Extracted: /tha_rules/VRT-policy.rules
Extracted: /usr/local/lib/snort_dynamicrules/web-activex.so
Extracted: /usr/local/lib/snort_dynamicrules/specific-threats.so
Extracted: /tha_rules/VRT-imap.rules
Extracted: /tha_rules/VRT-web-attacks.rules
Extracted: /tha_rules/VRT-file-flash.rules
Extracted: /usr/local/lib/snort_dynamicrules/icmp.so
Extracted: /tha_rules/VRT-nntp.rules
Extracted: /usr/local/lib/snort_dynamicrules/p2p.so
Extracted: /tha_rules/VRT-dos.rules
Extracted: /tha_rules/VRT-finger.rules
Extracted: /tha_rules/VRT-phishing-spam.rules
Extracted: /tha_rules/VRT-server-mysql.rules
Extracted: /tha_rules/VRT-oracle.rules
Extracted: /tha_rules/VRT-server-iis.rules
Extracted: /tha_rules/VRT-app-detect.rules
Extracted: /tha_rules/VRT-policy-multimedia.rules
Extracted: /tha_rules/VRT-pop2.rules
Extracted: /tha_rules/VRT-bad-traffic.rules
Extracted: /tha_rules/VRT-web-cgi.rules
Reading rules...
Generating Stub Rules....
Generating shared object stubs via:/usr/bin/snort -c /etc/nsm/templates/snort/snort.conf --dump-dynamic-rules=/tmp/tha_rules/so_rules/
An error occurred: WARNING: ip4 normalizations disabled because not inline.

An error occurred: WARNING: tcp normalizations disabled because not inline.

An error occurred: WARNING: icmp4 normalizations disabled because not inline.

An error occurred: WARNING: ip6 normalizations disabled because not inline.

An error occurred: WARNING: icmp6 normalizations disabled because not inline.

Dumping dynamic rules...
Dumping dynamic rules for Library web-activex 1.0.1
Dumping dynamic rules for Library web-client 1.0.1
Dumping dynamic rules for Library netbios 1.0.1
Dumping dynamic rules for Library exploit 1.0.1
Dumping dynamic rules for Library dos 1.0.1
Dumping dynamic rules for Library misc 1.0.1
Dumping dynamic rules for Library imap 1.0.1
Dumping dynamic rules for Library smtp 1.0.1
Dumping dynamic rules for Library web-iis 1.0.1
Dumping dynamic rules for Library chat 1.0.1
Dumping dynamic rules for Library icmp 1.0.1
Dumping dynamic rules for Library nntp 1.0.1
Dumping dynamic rules for Library p2p 1.0.1
Dumping dynamic rules for Library snmp 1.0.1
Dumping dynamic rules for Library multimedia 1.0.1
Dumping dynamic rules for Library bad-traffic 1.0.1
Dumping dynamic rules for Library web-misc 1.0.1
Dumping dynamic rules for Library specific-threats 1.0.1
Finished dumping dynamic rules.
Done
Reading rules...
Reading rules...
Reading rules...
Cleanup....
removed 138 temporary snort files or directories from /tmp/tha_rules!
Processing /etc/nsm/pulledpork/enablesid.conf....
Modified 0 rules
Done
Processing /etc/nsm/pulledpork/dropsid.conf....
Modified 0 rules
Done
Processing /etc/nsm/pulledpork/disablesid.conf....
Disabled 138:5
Disabled 3:14263
Disabled 3:7019
Disabled 119:31
Disabled 120:3
Disabled 119:19
Disabled 119:28
Disabled 119:15
Disabled 120:6
Disabled 119:33
Disabled 129:15
Disabled 129:5
Disabled 129:14
Disabled 145:2
Disabled 141:2
Disabled 140:2
Disabled 140:3
Disabled 140:4
Disabled 140:10
Disabled 140:8
Disabled 124:10
Disabled 1:23111
Disabled 1:20691
Disabled 1:20758
Disabled 1:23102
Disabled 1:25976
Disabled 1:20692
Disabled 1:25977
Disabled 1:25975
Disabled 138:6
Disabled 138:3
Disabled 138:2
Modified 32 rules
Done
Modifying Sids....
Done!
Setting Flowbit State....
Enabled 43 flowbits
Done
Writing /etc/nsm/rules/downloaded.rules....
Done
Writing /etc/nsm/rules/so_rules.rules....
Done
Generating sid-msg.map....
Done
Writing /etc/nsm/rules/sid-msg.map....
Done
Writing /var/log/sid_changes.log....
Done
Rule Stats....
New:-------330
Deleted:---22765
Enabled Rules:----4423
Dropped Rules:----0
Disabled Rules:---15020
Total Rules:------19443
Done
Please review /var/log/sid_changes.log for additional details
Fly Piggy Fly!

and

command: "grep -v "ET\ " /etc/nsm/rules/downloaded.rules | wc -l" resulted in 515 entries. which I expected.

command: cat sid-msg.map| grep "Phrovon"

results in

28244 || MALWARE-CNC WIN.Trojan.Phrovon outbound connection attempt || url,virustotal.com/en/file/d4fdedbe5891a13bd83b9d90f39a951ffa2f144df0b5d0ed613f7a107e6da1ad/analysis/1380571953/

so, now the rules are in place.

[Doug Burks]

Is it possible you had set LOCAL_NIDS_RULE_TUNING in
/etc/nsm/securityonion.conf (that would configure PulledPork to NOT
pull rules from the Internet)?
http://securityonion.blogspot.com/2013/06/new-securityonion-rule-update-package_25.html

What is the output of the following?
sudo rule-update

[Lonejeeper]

I have yet to peruse the rulesets to see where they overlap, but it's on the list. I don't enable all rules from both, but have tuned to our environment for the obvious rules.

my securityonion.conf:

ENGINE=snort
DAYSTOKEEP=50
ELSA=YES
OSSEC_AGENT_ENABLED=yes
BRO_ENABLED=yes
SNORBY_ENABLED=yes
XPLICO_ENABLED=yes

I have re-enabled the ETPRO rule_url entry to see if the "sudo /usr/bin/pulledpork.pl -c /etc/nsm/pulledpork/pulledpork.conf -vvv" command was successful. It was. The output of said command included the VRT and ETPRO rules being downloaded and extracted.

Here is rule_update with both rule_url entries in place:

Backing up current local_rules.xml file.
Cleaning up local_rules.xml backup files older than 3 days.
Backing up current downloaded.rules file before it gets overwritten.
Cleaning up downloaded.rules backup files older than 3 days.
Backing up current local.rules file before it gets overwritten.
Cleaning up local.rules backup files older than 3 days.
Running PulledPork.

http://code.google.com/p/pulledpork/
_____ ____
`----,\ )
`--==\\ / PulledPork v0.6.1 the Smoking Pig <////~
`--==\\/
.-~~~~-.Y|\\_ Copyright (C) 2009-2011 JJ Cummings
@_/ / 66\_ cumm...@gmail.com
| \ \ _(")
\ /-| ||'--' Rules give me wings!
\_\ \_\\
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Prepping rules from snortrules-snapshot-2953.tar.gz for work....

Done!
Prepping rules from etpro.rules.tar.gz for work....
Done!

Reading rules...
Generating Stub Rules....

Done
Reading rules...
Reading rules...
Reading rules...

Processing /etc/nsm/pulledpork/enablesid.conf....
Modified 0 rules
Done
Processing /etc/nsm/pulledpork/dropsid.conf....

Modified 5 rules
Done
Processing /etc/nsm/pulledpork/disablesid.conf....
Modified 1693 rules

Done
Modifying Sids....
Done!
Setting Flowbit State....

Enabled 96 flowbits
Enabled 1 flowbits
Enabled 1 flowbits

Done
Writing /etc/nsm/rules/downloaded.rules....
Done
Writing /etc/nsm/rules/so_rules.rules....
Done
Generating sid-msg.map....
Done
Writing /etc/nsm/rules/sid-msg.map....
Done
Writing /var/log/sid_changes.log....
Done
Rule Stats....

New:-------1
Deleted:---316
Enabled Rules:----23228
Dropped Rules:----5
Disabled Rules:---18709
Total Rules:------41942

Done
Please review /var/log/sid_changes.log for additional details
Fly Piggy Fly!

comparing these results against the previous scheduled run in /var/log/nsm/pulledpork.log shows that the recent rule_update has several thousand new rules since UTC 700.

Tue Oct 22 07:01:01 UTC 2013
Backing up current local_rules.xml file.
Cleaning up local_rules.xml backup files older than 3 days.
Backing up current downloaded.rules file before it gets overwritten.
Cleaning up downloaded.rules backup files older than 3 days.
Backing up current local.rules file before it gets overwritten.
Cleaning up local.rules backup files older than 3 days.
Running PulledPork.

http://code.google.com/p/pulledpork/
_____ ____
`----,\ )
`--==\\ / PulledPork v0.6.1 the Smoking Pig <////~
`--==\\/
.-~~~~-.Y|\\_ Copyright (C) 2009-2011 JJ Cummings
@_/ / 66\_ cumm...@gmail.com
| \ \ _(")
\ /-| ||'--' Rules give me wings!
\_\ \_\\
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Prepping rules from snortrules-snapshot-2953.tar.gz for work....

Done!
Prepping rules from etpro.rules.tar.gz for work....
Done!

Reading rules...
Generating Stub Rules....

Done
Reading rules...
Reading rules...
Reading rules...

Processing /etc/nsm/pulledpork/enablesid.conf....
Modified 0 rules
Done
Processing /etc/nsm/pulledpork/dropsid.conf....

Modified 5 rules
Done
Processing /etc/nsm/pulledpork/disablesid.conf....
Modified 1692 rules

Done
Modifying Sids....
Done!
Setting Flowbit State....

Enabled 96 flowbits
Enabled 1 flowbits
Enabled 1 flowbits

Done
Writing /etc/nsm/rules/downloaded.rules....
Done
Writing /etc/nsm/rules/so_rules.rules....
Done
Generating sid-msg.map....
Done
Writing /etc/nsm/rules/sid-msg.map....
Done
Writing /var/log/sid_changes.log....
Done
Rule Stats....

New:-------1
Deleted:---1
Enabled Rules:----23171
Dropped Rules:----5
Disabled Rules:---18702
Total Rules:------41878

[Doug Burks]

On Tue, Oct 22, 2013 at 10:47 AM, Lonejeeper <rockc...@gmail.com> wrote:
> comparing these results against the previous scheduled run in /var/log/nsm/pulledpork.log shows that the recent rule_update has several thousand new rules since UTC 700.

I'm not sure I understand what you're saying here. It looks to me
like the rule-update you just ran resulted in 41,942 rules whereas
this morning it was 41,878 rules (a difference of only 64):

Rule Stats....
New:-------1
Deleted:---316
Enabled Rules:----23228
Dropped Rules:----5
Disabled Rules:---18709
Total Rules:------41942
Done
Please review /var/log/sid_changes.log for additional details
Fly Piggy Fly!

Tue Oct 22 07:01:01 UTC 2013

<snip>

Rule Stats....
New:-------1
Deleted:---1
Enabled Rules:----23171
Dropped Rules:----5
Disabled Rules:---18702
Total Rules:------41878

In any case, 23K enabled rules is way too many. Please only enable
the rules that you really need.

----
Doug Burks
http://securityonion.blogspot.com

[Lonejeeper]

>
> I'm not sure I understand what you're saying here. It looks to me
>
> like the rule-update you just ran resulted in 41,942 rules whereas
>
> this morning it was 41,878 rules (a difference of only 64):
>

Ha! Me either, no idea where my head went on that one. My point is that it appears to have gotten additional rules. I'm going to wait until the VRT and ETPRO guys ship out some new rules and verify that rule-update is grabbing them. I believe the latest set is incorporated into SO from the manual pulledpork run. I don't feel like anything was 'fixed' at this point, nor do I understand what is going wrong.

>
> In any case, 23K enabled rules is way too many. Please only enable
>
> the rules that you really need.
>

That's interesting. I've been disabling rules that have generated hits that were irrelevant, and the policy rules have been heavily tweaked, and I've got whole categories disabled where appropriate.

I'm at a university, where we only really have control over staff machines, so I've got a pretty wide range of 'things' doing 'stuff'... my take is that I don't know what sort of attack we might see against something the CIS department put up so I'd want a wide variety of rules in place.

There's probably rules in place for software we'd never run, is it common for folks to comb through all the rules and disable them piecemeal like that? Where could I go to learn how to better manage this, is there a resource you could recommend?

Thanks for all your time and efforts Doug.

[Doug Burks]

23K rules is going to result in at least two issues:

1. high sensor CPU causing dropped packets

2. high numbers of alerts overwhelming analysts

Whether it's dropped packets or overwhelmed analysts, the net result
of either of these is missed attacks.

I'd probably start with *just* VRT or *just* ETPRO (not both) and
disable anything unnecessary until you get down to about 5K sigs.
Once you have a good minimal ruleset that's designed to be actionable
for your environment and you're able to process all of your alerts
every single day, then consider looking through the other ruleset for
a small number of signatures that you might want to add.

[Lonejeeper]

Excellent. It's actually "analyst", being just me.

Thanks Doug!