Re: [security-onion] Elsa is only showing what looks like local information and bro information

[Matt Gregory]

Hi Casper,

Querying class=none will necessarily return a lot of localhost records and shouldn't return any Bro records, since all the Bro records are of a specific class (e.g., class=BRO_CONN, class=BRO_HTTP).  Try a query for just an IP address on your monitored network that you know has generated a variety of traffic - you should get a variety of records such as BRO_CONN, BRO_HTTP, and BRO_DNS.

Out of the box, ELSA on Security Onion parses Snort logs and the most significant Bro logs (there are a few Bro records I believe it doesn't by default parse out, but you can still search by keyword even if a keyword is not associated with a specific field like "class" or "program")

Matt

On Sun, Apr 21, 2013 at 5:54 AM, <offe...@gmail.com> wrote:

Hi all,

So my ELSA stopped working.. thanks to scott i found the solution for that, had to restart my sphinx service.

So after doing that i went to my ELSA interface and starting searching for all kind of things with no joy..

Then i googled and looked through this site and found a post where you could test by searching for class=none.

This gave me some results but they all looked local. Like i had a bunch of 127.0.0.1 messengers and a few, very few Bro alerts.

I have not tuned my IDS yet so im seeing like 15000 or so alerts in snorby so i would asume these would show. I tried different IP that i know have had trafic and alerts on them but no joy there.

To me it looks like ELSA isn't getting the information or rather all the information.

Any ideas what is wrong?

My setup is a server/sensor topology im not really sure if you want a sudo sostat output from which server you want it from, the server or the sensor.

It is my understanding that ELSA is the "pick its brains" feature of SO? So all the information that is kept in the database can be searched on from ELSA, right?

Sincerely

Casper

--
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at http://groups.google.com/group/security-onion?hl=en-US.
For more options, visit https://groups.google.com/groups/opt_out.

[Heine Lysemose]

Hi Casper

What I usually do to test ELSA is to browse a few sites (www.google.com, www.dr.dk, etc) wait a minute to let ELSA load the data.
Then from the ELSA website choose BRO_CONN any from the Add Term drop down and search.

class=BRO_CONN

This should, if things are working, get you results. From here you can click on sites in the "Quick search" menu.

Regards,
Lysemose

[Martin Holste]

If the Bro logs you're looking for are on the same box as ELSA, then searching host=127.0.0.1 should present all logs.  Adding "groupby:hour" (or using the "Report On" button and selecting class "ANY" and "hour") should give you a good look at the influx of traffic over time.  Switching to "groupby:class" will give you a breakdown by class instead.

[Martin Holste]

It is a server/sensor topology, but if the logs were generated by the same host that received them, then they would show up from any query as host=127.0.0.1, because the query will be searching from the perspective of the individual sensors.

On Mon, Apr 22, 2013 at 4:38 AM, <offe...@gmail.com> wrote:

Hi all and thanks for the quick repsons...

@Matt

I did try to query an IP which i new recived allot of traffic and errors.. I belive I mentioned that in my OP aswell. So something is wrong...

@Heine
I did try to search for www.google.dk and google.dk with no results. This Server monitors the edge of my firewall along with 10 people so i assume there MUST be some information but i recived none.

@Martin
Its a server/sensor topology so i dont't think that apply here? or am i wrong?

/Casper

[Heine Lysemose]

What if you put this in your seacrch limit:1000

 

class=BRO_CONN limit:1000

 

/Heine

On Tue, Apr 23, 2013 at 2:43 PM, <offe...@gmail.com> wrote:

Also just found this in the top right corner of the ELSA webinterface

1 node(s) with 90126.0 logs indexed and 187958.0 archived

So to me it looks like my webinterface can't look up in the database correctly somehow.

I mean it says i have 187.958 archived logs and 90126 logs indexed and i only get about 100 events in the searches i posted before..

[Eric Ooi]

From the ELSA FAQ:

http://code.google.com/p/enterprise-log-search-and-archive/wiki/FAQ

How do I get more than 100 results returned?

Generally speaking you should filter your result to avoid having so many results returned. This can be done either with the hyphen-term syntax -term1 -term2 or adding in "must" conditionals +mustbe1 +mustbe2. This will allow you to spend less time scrolling through your result set looking for something of interest. If you can't filter your search like this, you can use the limit keyword to return an arbitrary amount of records with limit:1000. Up to 9999 results can be returned this way into the browser. More than 9999 will go into "bulk" mode in which the results are stored on the web server as a flat file and must be manually downloaded. An email is sent with a link to that download file upon completion of the search.

[Martin Holste]

You were getting good results back for host:127.0.0.1, so the next step is to see what logs you're getting there.  Add a groupby:class to your host:127.0.0.1 to see the unique classes coming in.  Hopefully, one of the classes there will be a Bro classes, and then you can drill-down by clicking that class.  You may also want to do a groupby:hour to see the rate at which they're coming in.

On Tue, Apr 23, 2013 at 1:14 PM, <offe...@gmail.com> wrote:

Hi guys and thanks for replying.

I don't think its a problem of how many search i request.. I tried your limit:1000 with both www.google.dk www.google.com I even tried it with the sig_msg:trojan which i find on you tube and i get nothing..

anyway to reinstall ELSA or something? I feels like it doesn't have all the "hooks" needed in the database...

[Martin Holste]

Do host:127.0.0.1 groupby:class to see all classes.  Hopefully there will be some Bro classes there.  You don't need the limit.

On Wed, Apr 24, 2013 at 10:22 AM, <offe...@gmail.com> wrote:

Hi Martin

I did this, im not sure if thats what you mean? i'm still green on this

limit:1000 host:127.0.0.1 class=BRO_CONN

And it returned nothing

[Heine Lysemose]

Try

 

host:127.0.0.1 groupby:class

host:127.0.0.1 groupby:hour

 

/Lysemose

On Wed, Apr 24, 2013 at 4:22 PM, <offe...@gmail.com> wrote:

Hi Martin

I did this, im not sure if thats what you mean? i'm still green on this

limit:1000 host:127.0.0.1 class=BRO_CONN

And it returned nothing

[Heine Lysemose]

Hi

 

Are your box receiving any traffic?

Any new data in Sguil, Squert, Snorby?

 

/Lysemose

[Martin Holste]

That indicates that Bro logs from the local machine aren't making it in to syslog, but others are.  What do you get for these queries:

classification groupby:class

classification groupby:host

tcp groupby:class

tcp groupby:host

On Wed, Apr 24, 2013 at 12:39 PM, <offe...@gmail.com> wrote:

Heine and Martin

now i get it!!! :)

First of, i'm getting tons of traffic daily.. I haven't gotten to tune my IDS yet..

The host:127.0.0.1 groupby:class gave me the following:

I see a column diagram, if thats what it is called, where the X axis has 3 values

NONE
SSH_Login
SSH_Logout

NONE has 16002 which must be events?
SSH_Login has 12
SSH_Logout has 12

When i click the column for NONE i get a detailed report of what it contained. At a glans they all look to be 127.0.0.1 based.

I asume you both were hoping  to see some IDS alerts and such but it doesn't seem to be so. It all seems to be local...

I'm getting a little desperate here because my CEO expect us to reach this milestone the first of May...

/Casper

[Martin Holste]

Ok, so none of the Bro logs are making it in.  Is this the only SecurityOnion sensor?  A good way to get a feel for what's coming in is to run a tcpdump on the management interface "like eth0" on port 514 to see what's being sent.  Local Bro logs would not show up on the network, though, they will be in the Bro logs directory.

On Wed, Apr 24, 2013 at 4:58 PM, <offe...@gmail.com> wrote:

Hey Martin,

classification groupby:class - No records

classification groupby:host - No records

tcp groupby:class - 3 None class hits. Three OSSEC agent keepalives with 127.0.0.1 as host

tcp groupby:host - The same as groupby:class, actually excatly the same...

Thanks

[Martin Holste]

Ok, so to sanity check that you're actually receiving data off the span port or tap (unless you know for a fact you are), just run:

tcpdump -i eth0 -n -c 20

That should spit out 20 packets being sniffed on the monitoring interface.  If that's true, then the next step is to verify that Bro is creating files.  I'm not sure what directory Bro logs go in on SecurityOnion, but you can find them for sure with:

find / -name weird.log

Look at the directory you find that in and check out files like conn.log, which should have a lot of entries.  If they do, that means Bro is working, so you need to find out why the flat-file logs aren't making it into ELSA.  For that, look at /nsm/elsa/data/elsa/log/node.log, go to the bottom of the file (using less), and then search backwards until you see a message about an index being loaded.  That will prove that ELSA is loading logs into the index.  Also look for any errors there.

On Wed, Apr 24, 2013 at 11:18 PM, <offe...@gmail.com> wrote:

Hi Martin,

Yes currently this is the only sensor...

just to make my process clear.

I have a Sensor01 and a server

When i use elsa i go to my server and pull the webinterface from there, that correct right? I suppose the server then look in elsa on the sensor..

On my sensor i have eth0 and eth1. Eth0 is my monitoring interface and eth1 is my mangement interface.

So what you are suggesting is to run tcpdump on my ETH1 mangement interface? The reason im asking is because my monitoring interface is ETH0 where i would assume the traffic would come to, so why not tcpdump that?

Currently i don't know how to use TCPDUMP but i guess i can google that faily easily

[Martin Holste]

Ok, so at least 463 logs made into that particular index, so ELSA is receiving logs just fine and it's apparently a matter of getting them from Bro.  What do the sizes of the Bro logs look like as written by Bro?

On Sun, Apr 28, 2013 at 6:21 PM, <offe...@gmail.com> wrote:

Studding the logs a little more or this is the second "grouped" hits with the word index in it:

* TRACE [2013/04/28 08:53:02] /opt/elsa/node/Indexer.pm (1523) Indexer::_sphinx_index 4310 ran cmd: /usr/bin/indexer --config /etc/sphinxsearch/sphinx.conf --rotate temp_20 2>&1
* INFO [2013/04/28 08:53:02] /opt/elsa/node/Indexer.pm (1552) Indexer::_sphinx_index 4310 Indexed temp_20 with 463 rows in 0.05735 seconds (8072.94370 rows/sec)
* TRACE [2013/04/28 08:53:02] /opt/elsa/node/Indexer.pm (1231) Indexer::_get_lock 4310 Locked directory
* TRACE [2013/04/28 08:53:03] /opt/elsa/node/Indexer.pm (1425) Indexer::index_records 4310 Unlocked indexes between 16578900 and 16579362
* TRACE [2013/04/28 08:53:03] /opt/elsa/node/Indexer.pm (1252) Indexer::_release_lock 4310 Unlocked directory
* TRACE [2013/04/28 08:53:03] /opt/elsa/node/Indexer.pm (2086) Indexer::record_host_stats 4310 Only found 1 hosts in temp_19

And here is the third:

* DEBUG [2013/04/28 08:53:02] /opt/elsa/node/Indexer.pm (1358) Indexer::index_records 4310 Indexing rows from table syslog_data.syslogs_index_10607389
* DEBUG [2013/04/28 08:53:02] /opt/elsa/node/Indexer.pm (1386) Indexer::index_records 4310 Data table info: 462, 1367139119, 1367139178
* DEBUG [2013/04/28 08:53:02] /opt/elsa/node/elsa.pl (146) main:: 30791 Starting process_batch
* TRACE [2013/04/28 08:53:02] /opt/elsa/node/elsa.pl (373) main::_fork_livetail_manager 30791 Forked livetail manager to pid 4315 with filename /nsm/elsa/data/elsa/tmp/buffers//1367139182
* TRACE [2013/04/28 08:53:02] /opt/elsa/node/elsa.pl (388) main::__ANON__ 4313 Livetail manager received TERM signal
* DEBUG [2013/04/28 08:53:02] /opt/elsa/node/elsa.pl (273) main::_process_batch 30791 Finished job process_batch with cache hits: 0 and 0 new programs
* TRACE [2013/04/28 08:53:02] /opt/elsa/node/elsa.pl (293) main::_process_batch 30791 Ending child tail manager 4315
* DEBUG [2013/04/28 08:53:02] /opt/elsa/node/elsa.pl (150) main:: 30791 Processed 0 records
* TRACE [2013/04/28 08:53:02] /opt/elsa/node/Indexer.pm (1252) Indexer::_release_lock 4310 Unlocked directory
* DEBUG [2013/04/28 08:53:02] /opt/elsa/node/Indexer.pm (1404) Indexer::index_records 4310 Inserted into indexes: 20, 1367139119, 1367139178, 16578900, 16579362, syslog_data.syslogs_index_10607389, temporary, 4310
* DEBUG [2013/04/28 08:53:02] /opt/elsa/node/Indexer.pm (1519) Indexer::_sphinx_index 4310 output: Sphinx 2.0.4-release (r3135)
Copyright (c) 2001-2012, Andrew Aksyonoff
Copyright (c) 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)