Towards Elastic on Security Onion: Technology Preview 3 (TP3)

[Doug Burks]

http://blog.securityonion.net/2017/07/towards-elastic-on-security-onion.html

--
Doug Burks

[Audrius J]

Hi Doug,

Just installed new release from scratch and it seems that I have some problems.
Logstash generates a lot of errors like this:
[2017-08-01T15:35:10,023][ERROR][logstash.filters.rest ] Error in Rest filter {:request=>[:get, "http://domainstats:20000/domain/creation_date/mesh.ad.jp", {}], :json=>false, :code=>nil, :body=>nil, :client_error=>#<Manticore::StreamClosedException: Could not read from stream: Read timed out>}

Disabling domainstats, resolves issues.
It seems, that a system can't resolve domainstats...

Regards,
Audrius

[Doug Burks]

Hi Audrius,

I just tested using the instructions on the blog post and it seems to
be working fine for me. In theory, when /usr/sbin/so-elastic-start
starts the logstash container it should link it to the domainstats
container which should then allow the logstash container to resolve
the hostname "domainstats". Are you sure all of your Docker images
were downloaded successfully and able to start?

> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.



--
Doug Burks

[Audrius J]

Hi Doug,

Installed it on other server and tested again. It seems, what it is woking just fine now.
Thanks!

Audrius

[Kevin Branch]

I recently upgraded a fairly heavy duty test server from TP2 to TP3.  Initially Kibana showed events, but due to the large volume of events on this test server (over 100eps, mostly bro_conn), within a few minutes new events ceased showing up in Kibana.  This is because Logstash was running over 100% on the CPU and throwing lots of grok timeout errors which caused the persistent queue to fill up and jam the pipeline to Elasticsearch.

Increasing heap memory for Logstash and Elasticsearch was needed but not enough to resolve the issue:

/etc/nsm/securityonion.conf

LOGSTASH_HEAP="4g" (probably overkill)

ELASTICSEARCH_HEAP="8g"

I also needed to increase pipeline.workers in /etc/logstash/logstash.yml to 8.  Since that change, logstash has been able to use enough CPU to keep up with the incoming log stream.

One other thing I noticed in so-elastic-status was that the elastalert container was using nearly 100% CPU and have a very high NET I/O volume not too far from what Logstash and Elasticsearch were doing.  Looking at the two frequency rules, I saw they were not using use_count_query which means massive data was getting pulled from Elasticsearch to ElastAlert just for the purpose of ElastAlert to count records.  I added the following to the two ElastAlert rules, which I believe makes ElastAlert just ask Elasticsearch for a record count instead of a data dump, and now the so-elastalert container is humming along with less than 1% cpu and trivially low NET I/O.  

/etc/elastalert/rules/bro_conn.yaml

use_count_query: true

doc_type: 'bro_conn'

/etc/elastalert/rules/ids.yaml

use_count_query: true

doc_type: 'snort'

I confirmed that both ElastAlert rules are still firing pretty much every minute as expected with the current frequency threshold of one event per minute.

Also, I am really impressed with how so-freqserver and so-domainstats are being used by Logstash to enrich the bro_dns records for bettern DNS security analytics.  Something along this line was already on my to-do list and this already appears to be exceeding my expectations in this area.  I really need to sharpen my Kibana ninja skills for better interaction with this new data.

Thanks for the awesome work!

Kevin

[Wes Lambert]

Thanks for the feedback, Kevin!

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

[Brodie Mather]

I am having an issue when running setup after configuring the interfaces.

It is stuck saying "Please wait while configuring Elastic"

I have given it well over an hour and it still did not complete.

What can I do to remedy this situation?

Thanks,
Brodie

[Doug Burks]

Hi Brodie,

I just tested and it worked fine for me.

Are you able to send a copy of /tmp/sosetup.log.$RANDOM (where $RANDOM
is a string of random letters and numbers)?

Also, please run the following command:

sudo sostat-redacted
There will be a lot of output, so you may need to increase your
terminal's scroll buffer OR redirect the output of the command to a
file:

sudo sostat-redacted > sostat-redacted.txt 2>&1
sostat-redacted will automatically redact any IPv4/IPv6/MAC addresses,
but there may be additional sensitive info that you still need to
redact manually.

Attach the output to your email in plain text format (.txt) OR use a
service like http://pastebin.com.

> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups "security-onion" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.

> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.

--
Doug Burks

[Brodie Mather]

Doug,

I just sent an email to security onion one, I hope that was where you wanted it sent.

If not, please let me know.

Thanks,
Brodie

[Doug Burks]

Hi Brodie,

I'm not sure what you mean by "security onion one". Are you able to
attach those files to your reply in this email thread?

[Brodie Mather]

Doug,

Sorry, I misunderstood. I thought you meant to send an actual email, not just post here.

Here is the sostat-redacted file.

I had to restart the machine because it was stalling on the sostat right before the elasticsearch part, so I believe that caused the logs to disappear.

Thanks,
Brodie

[Doug Burks]

Replies inline.

On Mon, Aug 14, 2017 at 11:50 AM, 'Brodie Mather' via security-onion
<securit...@googlegroups.com> wrote:
> Doug,
>
> Sorry, I misunderstood. I thought you meant to send an actual email, not just post here.

This is a Google Group, which can be used as a mailing list or a web
forum. So you can use whatever is most convenient for you, email or
web interface.

> Here is the sostat-redacted file.
>
> I had to restart the machine because it was stalling on the sostat right before the elasticsearch part, so I believe that caused the logs to disappear.

If you restarted the machine while Setup was still running, then your
system is in an inconsistent state and the sosetup.log in /tmp is most
likely gone.

Please try a fresh installation. If the same thing happens again,
please get a copy of /tmp/sosetup.log.$RANDOM (where $RANDOM is a
string of random letters and numbers) before restarting the machine.

Thanks!


--
Doug Burks

[Brodie Mather]

Doug,

After a fresh install I get the same situation.

Here is the attached setup log.

Thanks,
Brodie

[Doug Burks]

Based on the end of sosetup.log, it looks like so-elastic-start is
waiting on Elasticsearch:
https://github.com/Security-Onion-Solutions/elastic-test/blob/master/usr/sbin/so-elastic-start#L98

So the next question would be: what is Elasticsearch doing? Can you
provide your /var/log/elasticsearch/docker-cluster.log?

[Brodie Mather]

It is attached.

[Doug Burks]

Looks like Elasticsearch has started and it thinks it's listening on
ports 9200 and 9300. What's the output of the following?
curl http://localhost:9200

On Mon, Aug 14, 2017 at 4:02 PM, 'Brodie Mather' via security-onion
<securit...@googlegroups.com> wrote:
> It is attached.

[Brodie Mather]

Doug,

It took a long time to respond and finally came back with:

"curl: (56) Recv failure: Connection reset by peer"

Thanks,
Brodie

[Doug Burks]

Is this a physical machine or VM?

Does it have anything installed other than Security Onion?

How did you install/configure the machine?

What is the output of the following?
sudo service docker restart
sudo so-elastic-restart

[Brodie Mather]

Replies are inline

On Tuesday, August 15, 2017 at 11:39:29 AM UTC-5, Doug Burks wrote:
> Is this a physical machine or VM?

> It is a VM

> Does it have anything installed other than Security Onion?

> The only thing installed since the VM was created was securtiyonion and then I followed the steps for TP 3.

> How did you install/configure the machine?

> The machine has the specifications required from the blog post of TP3.

> What is the output of the following?
> sudo service docker restart

docker stop/waiting
docker start/running, process processID

> sudo so-elastic-restart
> This removed the exisiting containers (elasticsearch, logstash, freqserver, and domainstats)
Then it started the same containers and now it is stalled saying waiting for ElasticSearch.

Thanks

Brodie

[Doug Burks]

What is the output of the following?

curl http://172.17.0.4:9200

[Brodie Mather]

curl: (7) Failed to connect to 172.17.0.4 port 9200: Connection timed out

[Doug Burks]

This is really strange because if Elasticsearch is listening on port
9200, then you should be able to connect to it via Docker proxy (curl
http://localhost:9200) or directly to the Docker container (curl
http://172.17.0.4:9200).

What is the output of the following?

ps aux |grep proxy

Is it possible your normal LAN address space (eth0) is 172.x?

On Tue, Aug 15, 2017 at 3:11 PM, 'Brodie Mather' via security-onion
<securit...@googlegroups.com> wrote:
> curl: (7) Failed to connect to 172.17.0.4 port 9200: Connection timed out
>

[Brodie Mather]

cybele 7561 0.0 0.0 18936 2268 pts/2 S+ 19:37 0:00 grep --color=auto proxy
root 21813 0.0 0.0 50848 2932 ? Sl 16:53 0:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 10004 -container-ip 172.17.0.2 -container-port 10004
root 21941 0.0 0.0 50848 2824 ? Sl 16:53 0:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 20000 -container-ip 172.17.0.3 -container-port 20000
root 22060 0.0 0.0 108444 2864 ? Sl 16:53 0:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 9200 -container-ip 172.17.0.4 -container-port 9200
root 22207 0.0 0.0 34456 2820 ? Sl 16:53 0:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 6053 -container-ip 172.17.0.5 -container-port 6053
root 22234 0.0 0.0 34456 2904 ? Sl 16:53 0:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 6052 -container-ip 172.17.0.5 -container-port 6052
root 22251 0.0 0.0 34456 2908 ? Sl 16:53 0:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 6051 -container-ip 172.17.0.5 -container-port 6051
root 22262 0.0 0.0 34456 2892 ? Sl 16:53 0:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 6050 -container-ip 172.17.0.5 -container-port 6050

No, eth0 does not have a LAN address space of 172.x

[Doug Burks]

That looks correct. You might try the following to see if it makes
any difference:
- revert the VM to a clean snapshot
- install all updates using "sudo soup"
- reboot the VM
- then install TP3

[Brodie Mather]

The VM was brand new before installing securityonion and subsequently TP3. I do not have any snapshots because there were no prior states to the VM before trying the TP 3 install.

Does the TP 3 script/setup run sudo soup or no?

[Doug Burks]

No, the TP3 script does not automatically run "sudo soup". It
probably won't make a difference, but it's worth a shot since I'm
unable to duplicate your issue.

[Brodie Mather]

Tried to run the sudo soup and it stall while restarting docker containers

[Doug Burks]

Was this running "sudo soup" on your existing broken system? If so,
I'd recommend the following:

- destroy the VM
- create a new VM with a fresh installation of Security Onion
- run "sudo soup"
- reboot
- create a snapshot so that you can revert back to pre-TP3 status if necessary
- install TP3

If you still get the same issue, then try removing the following lines
from /usr/sbin/so-elastic-start and then reboot:

# Wait for ElasticSearch to come up, so that we can query for
version infromation
echo
echo -n "Waiting for ElasticSearch"
until $(curl --output /dev/null --silent --head --fail
http://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT"); do
echo -n '.'
sleep 1
done
echo
echo "Connection to ElasticSearch successful!"
echo
# Compare current version of ES with previously stored version.
CURRENT_KIBANA_VERSION=$(curl -s
"http://$ELASTICSEARCH_HOST:$ELASTICSEARCH_PORT" | jq .version.number
| sed 's/"//g')
STORED_KIBANA_VERSION=$(grep 'KIBANA_VERSION'
/etc/nsm/securityonion.conf | cut -d'=' -f2 )
if [ "$CURRENT_KIBANA_VERSION" != "$STORED_KIBANA_VERSION" ]; then
sed -i
"s/KIBANA_VERSION=.*/KIBANA_VERSION=$KIBANA_VERSION/"
/etc/nsm/securityonion.conf
/usr/sbin/so-elastic-configure-kibana -r > /dev/null 2>&1
fi

On Tue, Aug 15, 2017 at 5:31 PM, 'Brodie Mather' via security-onion
<securit...@googlegroups.com> wrote:
> Tried to run the sudo soup and it stall while restarting docker containers
>

[Brodie Mather]

Tried all of these steps.
Ran setup after the reboot finished. And the same issue the setup stalls
while saying "Please wait while configuring Elastic"

The log from tmp/ has a final message of "Configuring Kibana"

[Doug Burks]

If it says "Configuring Kibana", I think that would have come from
/usr/sbin/so-elastic-configure-kibana. Are you sure you removed all
the lines in /usr/sbin/so-elastic-start per my previous email?

What is the output of the following?

nslookup localhost
nslookup elasticsearch
nslookup so-elasticsearch

Again, I'm really shooting in the dark here since I'm not able to
duplicate this issue. Any chance you could try this on a different
machine and/or different network?

[Brodie Mather]

Replies Inline

On Wednesday, August 16, 2017 at 5:56:35 PM UTC, Doug Burks wrote:
> If it says "Configuring Kibana", I think that would have come from
> /usr/sbin/so-elastic-configure-kibana. Are you sure you removed all
> the lines in /usr/sbin/so-elastic-start per my previous email?
>
> What is the output of the following?
> nslookup localhost

Server: 184.182.233.53
Address: 184.182.233.53#53

Name: localhost
Address: 127.0.0.1

> nslookup elasticsearch
Server: 184.182.233.53
Address: 184.182.233.53#53

** server can't find elasticsearch: NXDOMAIN

> nslookup so-elasticsearch
cybele@ELK-TP3-virtual-machine:~$ nslookup so-elasticsearch
Server: 184.182.233.53
Address: 184.182.233.53#53

** server can't find so-elasticsearch: NXDOMAIN

>
> Again, I'm really shooting in the dark here since I'm not able to
> duplicate this issue. Any chance you could try this on a different
> machine and/or different network?

This virtual machine is being created on an ESXI server from vsphere.
Not sure I can move it to a different network either.
I had no problem with the TP 2 setup process on basically an identical VM.

[Doug Burks]

On Wed, Aug 16, 2017 at 3:12 PM, 'Brodie Mather' via security-onion
<securit...@googlegroups.com> wrote:
>> Again, I'm really shooting in the dark here since I'm not able to
>> duplicate this issue. Any chance you could try this on a different
>> machine and/or different network?
> This virtual machine is being created on an ESXI server from vsphere.
> Not sure I can move it to a different network either.
> I had no problem with the TP 2 setup process on basically an identical VM.

Are you able to test on a laptop using VMware or some other
virtualization software (either on your corporate network or your home
network)?

Lots of folks have tried TP3 and nobody else has reported this issue.


--
Doug Burks

[Doug Burks]

I'll also point out that, in the TP2 thread
(https://groups.google.com/d/topic/security-onion/pi2TP7DgGIE/discussion),
you mentioned you were getting errors like "unable to connect to
elasticsearch". That sounds very similar to the symptoms you are
experiencing with TP3. Is it possible that these elasticsearch
problems with both TP2 and TP3 are related to some local change on
your network or ESXI configuration on or about August 10?


--
Doug Burks

[Brodie Mather]

I am attempting to try to machine somewhere else. I do not handle any of the networking or the configuration for the ESXI so I will have to ask around and see if anything changed.

Thanks,
Brodie

[kxuan celtik]

Hi,

I have the same problem.

After many proof I have success if Guest Additions ( VirtualBox) are not installed.

[Konrad Uminski]

Doug, is there a timeline for this project?

[Doug Burks]

Hi Kxuan,

I'm not sure I understand. Are you saying that VirtualBox Guest
Additions were preventing Kibana from connecting to Elasticsearch?
Can you provide more information, please?


--
Doug Burks

[Doug Burks]

Hi Konrad,

This is a big project and it's very important that we get it right, so
we're not ready to commit to any timelines just yet. Look for more
information to be announced around the time of the Security Onion
Conference.

On Thu, Aug 24, 2017 at 8:56 AM, Konrad Uminski <konrad...@gmail.com> wrote:
> Doug, is there a timeline for this project?
>

[kxuan celtik]

Hi Doug,

No, sorry, I only say that the second Setup with VirtualGuesst doesn't finished.

But It is only one data more, but it doesn't have relation with other proofs.

If it data could be useful for someone, perfect.

[Doug Burks]

I'm not sure why VirtualBox Guest Additions would be interfering.
Could you try again and see if you can duplicate the issue? If so,
can you then provide a copy of /tmp/sosetup.log.*?


--
Doug Burks

[kxuan celtik]

Doug,

Another issue that it's possible more near it's that in all my wrong first installations I chose docker0 and eth0 as monitoring interface.

And in my successful installation VirtualGuesst neither docker0 interface ( in the first setup )

I'm remembering now.

No, I can't reproduce the issue, sorry. It is much for my time now and I deleted all VMs.

Ahh, in the TIP 3 could you edit and put there that at first you should run 'sudo soup'. That's is clear, no?

[Doug Burks]

Hi kxuan,

Replies inline.

On Sun, Aug 27, 2017 at 7:26 AM, kxuan celtik <kxuan....@gmail.com> wrote:
> Doug,
>
> Another issue that it's possible more near it's that in all my wrong first installations I chose docker0 and eth0 as monitoring interface.

I just tested and it does look like selecting docker0 causes a
problem. I'll make sure we exclude those interfaces from Setup.

> And in my successful installation VirtualGuesst neither docker0 interface ( in the first setup )
>
> I'm remembering now.
>
> No, I can't reproduce the issue, sorry. It is much for my time now and I deleted all VMs.
>
> Ahh, in the TIP 3 could you edit and put there that at first you should run 'sudo soup'. That's is clear, no?

I just tested a fresh installation of 14.04.5.2 with no "sudo soup"
and it seems to be working fine for me.


--
Doug Burks

[kxuan celtik]

Definitively Virtual Box Guess Additions don't have conflicts with ELK on SO.

I've been installed it and I don't have any problem.

[kxuan celtik]

Hi,

I want to have kibana accessible from out.

I edit kibana.yml, in host line I write: "MyIPofManageInterface"

And I've the next error:

{"type":"log","@timestamp":"2017-08-28T22:58:11Z","tags":["fatal"],"pid":1,"level":"fatal","message":"listen EADDRNOTAVAIL 10.10.10.250:5601","error":{"message":"listen EADDRNOTAVAIL 10.10.10.250:5601","name":"Error","stack":"Error: listen EADDRNOTAVAIL 10.10.10.250:5601\n at Object.exports._errnoException (util.js:1018:11)\n at exports._exceptionWithHostPort (util.js:1041:20)\n at Server._listen2 (net.js:1249:19)\n at listen (net.js:1298:10)\n at net.js:1408:9\n at _combinedTickCallback (internal/process/next_tick.js:83:11)\n at process._tickCallback (internal/process/next_tick.js:104:9)","code":"EADDRNOTAVAIL"}}

How could I access from other host to UI interfaces like kibana?

Opening 443 port I reached SO web page.

The problem is that so-kibana is stopped with this fatal error.

Thanks,

[Wes Lambert]

Are you referring to accessing the Kibana web interface from outside of the local host?

If so, you should only have to open port 443 in ufw (or using so-allow) and access the url via https://ipaddress/app/kibana.

You should not edit any Kibana config files.

Thanks,

Wes

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onion+unsubscribe@googlegroups.com.
To post to this group, send email to security-onion@googlegroups.com.

[kxuan celtik]

Hi Wes,

Yes, you are right.

Thanks

I think that it need a change, sorry.

[Daniel K]

Doug, will current SO in deployment be upgradable to the soon to come version with ELK? If not, do you have any estimate when current ELK version will be available?

/Daniel

On Friday, July 28, 2017 at 10:29:55 PM UTC+2, Doug Burks wrote:
> http://blog.securityonion.net/2017/07/towards-elastic-on-security-onion.html
>
> --
> Doug Burks

[Doug Burks]

Hi Daniel,

Yes, we do plan to allow you to upgrade your existing Security Onion
boxes to the Elastic stack. You can test this in a VM today using the
instructions in the TP3 blog post:
http://blog.securityonion.net/2017/07/towards-elastic-on-security-onion.html

> --
> Follow Security Onion on Twitter!
> https://twitter.com/securityonion
> ---
> You received this message because you are subscribed to the Google Groups "security-onion" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.

> Visit this group at https://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.

--
Doug Burks