Index vs. Archive

[senatorh...@gmail.com]

I really need some closure on the difference between Indexed and Archived data.

Right now my ELSA (running for 6 months) shows me 1 billion logs indexed and 1 billion logs archived. If that is the case, why would I ever want to search in the archive?

Does the archive hold more data than the index?

I just don't get the point search-wise.

Another thing that really has been bugging me is, on how to make a query of all dstip seen today using the bro.conn.

ELSA will usually crash/timeout if I set the limit value. It's not exactly a heavy query.

Cheers.

[Wes Lambert]

Having a certain percentage of logs archived uses much less disk space than having all of your logs indexed.  You can certainly make changes to your configuration where ELSA will have all logs indexed (/etc/elsa_node.conf), but you would need to ensure you can support the amount of storage required.  If you search past your index range in ELSA, it will search the archived logs.  Searching archived logs takes longer than searching indexed logs.

Your last point may be due to a number of factors specific to your system.  If you would like to dive further into that, please provide the output of sostat redacted for any server/sensors running ELSA, attaching as a plain text file or using a service like Pastebin.com.

Thanks,

Wes

--
You received this message because you are subscribed to the Google Groups "enterprise-log-search-and-archive" group.
To unsubscribe from this group and stop receiving emails from it, send an email to enterprise-log-search-and-archive+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[senatorh...@gmail.com]

Error that I always encounter:

Query failed, low-level ELSA problem such as config, communication, OS error.

[Wes Lambert]

As mentioned previously, please provide the output of sostat-redacted, attaching as a plain text, or using a service like Pastebin.com

Thanks,

Wes

On Mon, Oct 2, 2017 at 2:04 PM, <senatorh...@gmail.com> wrote:

Error that I always encounter:

Query failed, low-level ELSA problem such as config, communication, OS error.

--

[senatorh...@gmail.com]

Sorry Wes. I had to check line by line before posting it in public.

https://pastebin.com/20eSkDrr

1-2 days is never a problem for queries. It seems like the queries extending beyond that will trigger the previous posted error.

An extreme example of one of my queries could be one like the below, which includes transforms.

BRO.CONN groupby:dstip start:"01-01-2017" end:"today" | whois | sum(descr)

[Wes Lambert]

Trying to use transforms (like whois) on a very large set of information (like a query extending all the way back to January) can take up a lot of resources (due to the fact that you are searching archived logs, which is traditionally a much slower search, combined with another operation).

I would advise that you narrow your focus before trying to perform these types of queries, or adjust your indexed vs archived logs in /etc/elsa_node.conf .

Thanks,

Wes

--

[senatorh...@gmail.com]

Dear Wes. But all of my data is indexed according to the ELSA GUI?