In ELSA, Bro Notice query not showing cymru notices for demo pcap entries -- expected behavior?

[Jim Solderitsch]

I am following along with Doug's videos for Security Onion and just viewed the ELSA one: https://www.youtube.com/watch?v=d4rINH22MYo&index=7&list=PLMN5wm-C5YjyieO63g8LbaiWTSJRj0DBe

Toward the end, it shows a ELSA notice query where there are 5 notice types that occur with the demo pcap data that was processed.

One of them was a cymru notice and when I run the same pcaps through my instance of Security Onion and execute the ELSA sequence shown in the video, I only get 2 notice types and no cymru notices.

Has the automatic behavior of checking hashes with cymru been disabled in the default Security Onion configuration from the latest iso?

Can it be turned on for demonstration? I am teaching a class where it might be useful to have this effect occur.

Thanks

Jim

[Doug Burks]

Hi Jim,

Team Cymru Malware Hash Registry (MHR) matching should be turned on by
default. You can verify as follows:

grep detect-MHR /opt/bro/share/bro/site/local.bro
@load frameworks/files/detect-MHR

detect-MHR relies on sending DNS requests to Team Cymru, so have you
verified that your box is able to do so?

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com

[Jim Solderitsch]

Did the grep and the @load string comes back.

A dig command to cymru does not error out.

I notice that ELSA also reports no MD5 notices either. So if these are not there, then nothing will be sent to cymru.

I was able to go through almost all of the forensic analysis steps outlined in the Fall 2013 video sequence except for the Bro Notices query.

Odd.

Is there a bro setting for doing the MD5 computations?

Jim

On Wednesday, September 23, 2015 at 5:37:50 AM UTC-4, Doug Burks wrote:
> Hi Jim,
>
> Team Cymru Malware Hash Registry (MHR) matching should be turned on by
> default. You can verify as follows:
>
> grep detect-MHR /opt/bro/share/bro/site/local.bro
> @load frameworks/files/detect-MHR
>
> detect-MHR relies on sending DNS requests to Team Cymru, so have you
> verified that your box is able to do so?
>

[Doug Burks]

On Wed, Sep 23, 2015 at 3:37 PM, Jim Solderitsch <jsolde...@gmail.com> wrote:
> Did the grep and the @load string comes back.
>
> A dig command to cymru does not error out.
>
> I notice that ELSA also reports no MD5 notices either. So if these are not there, then nothing will be sent to cymru.
>
> I was able to go through almost all of the forensic analysis steps outlined in the Fall 2013 video sequence except for the Bro Notices query.
>
> Odd.
>
> Is there a bro setting for doing the MD5 computations?

Yes, it's right above the detect-MHR setting in
/opt/bro/share/bro/site/local.bro:

# Enable MD5 and SHA1 hashing for all files.
@load frameworks/files/hash-all-files

# Detect SHA1 sums in Team Cymru's Malware Hash Registry.
@load frameworks/files/detect-MHR

If your local.bro looks like that but you're definitely not getting
MD5sums in your Bro logs, then please send sostat output.

Please run the following command:

sudo sostat-redacted

There will be a lot of output, so you may need to increase your
terminal's scroll buffer OR redirect the output of the command to a
file:

sudo sostat-redacted > sostat-redacted.txt 2>&1

sostat-redacted will automatically redact any IPv4/IPv6/MAC addresses,
but there may be additional sensitive info that you still need to
redact manually.

Attach the output to your email in plain text format (.txt) OR use a
service likehttp://pastebin.com.

[Jim Solderitsch]

I replayed the same demo traffic and still do not see MD5 or cymru notices from bro via ELSA. I checked the bro config file and both options are enabled.

The sostat-redacted output is at: http://pastebin.com/syKVpJGB

This is a plain demo Security Onion instance that I configured earlier this week from scratch. Running as a single VM in VMware Fusion 7.

Thanks for the continued interest in my attempt to follow along with the videos.

Jim

On Thursday, September 24, 2015 at 6:49:55 AM UTC-4, Doug Burks wrote:

[Heine Lysemose]

Hi

Can you post the command you use to replay the traffic?

Also you are low on memory... You have the possibility to add some more to your VM, if possible double up.

Also try disabling some of the processes, https://github.com/Security-Onion-Solutions/security-onion/wiki/DisablingProcesses
prads
sancp_agent
pads_agent
http_agent

These processes writes it data to the Sguil database which is the same as Bro logs, https://github.com/Security-Onion-Solutions/security-onion/wiki/Best-Practices

Regards,
Lysemose

[Jim Solderitsch]

Yes, the commands I used were:

sudo tcpreplay -ieth1 -M10 /opt/samples/markofu/honeynet_suspicious-time.pcap
sudo tcpreplay -ieth1 -M10 /opt/samples/markofu/outbound.pcap
sudo tcpreplay -ieth1 -M10 /opt/samples/markofu/jackcr-challenge.pcap
sudo tcpreplay -ieth1 -M10 /opt/samples/markofu/netforensics_evidence0*

These are what Doug uses in the Demo video. I configured Security Onion with 2 processors and 3 GBs of RAM.

I will try your configuration changes and report back if they make a difference.

Jim

[Jim Solderitsch]

I did make all of the configuration changes that Lysemose suggested in his reply earlier. I re-ran the same packet captures through my demo VM after rebooting to make sure all the changes were in effect.

Still no MD5 notices and then no cymru notices either.

Can anyone confirm that the latest .iso when used to make fresh install leads to Bro Notice Types being seen for MD5 hashes?

Puzzled,

Jim

[Doug Burks]

Hi Jim,

I started with the latest ISO image (12.04.5.3) and ran the same
tcpreplay commands you did:

sudo tcpreplay -ieth1 -M10 /opt/samples/markofu/honeynet_suspicious-time.pcap
sudo tcpreplay -ieth1 -M10 /opt/samples/markofu/outbound.pcap
sudo tcpreplay -ieth1 -M10 /opt/samples/markofu/jackcr-challenge.pcap
sudo tcpreplay -ieth1 -M10 /opt/samples/markofu/netforensics_evidence0*

I got two notices in notice.log, one for a port scan and one for an
invalid SSL cert.

Bro doesn't generate a Notice for each and every MD5 hash that it
sees, but it should generate an entry in files.log. So I can log into
ELSA and click "Files - MIME Types" and see that I have 7
application/x-dosexec files (Windows EXEs). I can also go to "HTTP -
Sites Hosting EXEs" and see those same 7 Windows EXEs.

If you're specifically looking for Team Cymru Malware Hash Registry
(MHR) matches, then try the following tcpreplay:
sudo tcpreplay -ieth1 -M10 /opt/samples/*.pcap


On Tue, Sep 29, 2015 at 12:16 AM, Jim Solderitsch

[Jim Solderitsch]

OK, that's for confirming my experience. With the video sequence you did in 2013, Bro did manage to produce other notice types for the 4 pcap injections that were originally used so something isn't quite the same with the latest release. Not a big issue, but certainly not my expected outcome. When I did the same follow along in the Fall of 2014 with the Security Onion release that was current then, the videos and my personal experience were the same down to the md5 and team cymru notice types.

Some things change, some remain the same I guess.

Again, thanks for your interest.

Jim

On Tuesday, September 29, 2015 at 5:29:29 AM UTC-4, Doug Burks wrote:
> Hi Jim,
>
> I started with the latest ISO image (12.04.5.3) and ran the same
> tcpreplay commands you did:
>
> sudo tcpreplay -ieth1 -M10 /opt/samples/markofu/honeynet_suspicious-time.pcap
> sudo tcpreplay -ieth1 -M10 /opt/samples/markofu/outbound.pcap
> sudo tcpreplay -ieth1 -M10 /opt/samples/markofu/jackcr-challenge.pcap
> sudo tcpreplay -ieth1 -M10 /opt/samples/markofu/netforensics_evidence0*
>
> I got two notices in notice.log, one for a port scan and one for an
> invalid SSL cert.
>
> Bro doesn't generate a Notice for each and every MD5 hash that it
> sees, but it should generate an entry in files.log. So I can log into
> ELSA and click "Files - MIME Types" and see that I have 7
> application/x-dosexec files (Windows EXEs). I can also go to "HTTP -
> Sites Hosting EXEs" and see those same 7 Windows EXEs.
>
> If you're specifically looking for Team Cymru Malware Hash Registry
> (MHR) matches, then try the following tcpreplay:
> sudo tcpreplay -ieth1 -M10 /opt/samples/*.pcap
>
>
> On Tue, Sep 29, 2015 at 12:16 AM, Jim Solderitsch