kibana broken after update

[Kris Springer]

I ran 'sudo soup' and after it completed and rebooted I couldn't access Kibana. I can access Squert, but not Kibana.

I ran 'sudo sostat' and it says 'Logstash is running', 'Kibana is not running', 'ElastAlert is not running', 'Curator is not running'. It suggests to run 'sudo so-elastic-start', which I did. Running 'sudo sostat' again says 'Kibana is not running' but Curator and ElastAlert are now running.

Running 'sudo so-elastic-start' a second time results in Kibana getting started and running successfully. But, the Kibana webpage says 'Kibana server is not ready yet'.

What happened? All worked great until I ran 'sudo soup'.

[Kris Springer]

I ran 'sudo soup' on a second SO Server of ours and the exact same issues have occurred. Kibana is not working on a second server now.

[Doug Burks]

Hi Kris,

Have you checked the Kibana log for any additional clues?

/var/log/kibana/kibana.log

On Mon, May 20, 2019 at 1:33 PM Kris Springer <kspr...@innovateteam.com> wrote:

I ran 'sudo soup' on a second SO Server of ours and the exact same issues have occurred.  Kibana is not working on a second server now.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/efc89da7-89dd-4a5c-aae7-3eaa352038f9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Doug Burks
CEO
Security Onion Solutions, LLC

[Kris Springer]

Thanks Doug. It seems Server1 and Server2 are having slightly different issues. Here's the log tail of Server1 that I manually redacted my domain name.

{"type":"response","@timestamp":"2019-05-20T18:12:49Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/bundles/app/kibana/bootstrap.js","method":"get","headers":{"host":"127.0.0.1:5601","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36","dnt":"1","accept":"*/*","referer":"https://REDACTED/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","x-forwarded-for":"65.129.48.36","x-forwarded-host":"REDACTED","x-forwarded-server":"localhost","connection":"Keep-Alive"},"remoteAddress":"172.17.0.1","userAgent":"172.17.0.1","referer":"https://REDACTED/app/kibana"},"res":{"statusCode":200,"responseTime":5,"contentLength":9},"message":"GET /bundles/app/kibana/bootstrap.js 200 5ms - 9.0B"}
{"type":"response","@timestamp":"2019-05-20T18:12:49Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/bundles/commons.style.css","method":"get","headers":{"host":"127.0.0.1:5601","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36","dnt":"1","accept":"text/css,*/*;q=0.1","referer":"https://REDACTED/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","x-forwarded-for":"65.129.48.36","x-forwarded-host":"REDACTED","x-forwarded-server":"localhost","connection":"Keep-Alive"},"remoteAddress":"172.17.0.1","userAgent":"172.17.0.1","referer":"https://REDACTED/app/kibana"},"res":{"statusCode":200,"responseTime":12,"contentLength":9},"message":"GET /bundles/commons.style.css 200 12ms - 9.0B"}
{"type":"response","@timestamp":"2019-05-20T18:12:49Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/bundles/kibana.style.css","method":"get","headers":{"host":"127.0.0.1:5601","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36","dnt":"1","accept":"text/css,*/*;q=0.1","referer":"https://REDACTED/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","x-forwarded-for":"65.129.48.36","x-forwarded-host":"REDACTED","x-forwarded-server":"localhost","connection":"Keep-Alive"},"remoteAddress":"172.17.0.1","userAgent":"172.17.0.1","referer":"https://REDACTED/app/kibana"},"res":{"statusCode":200,"responseTime":2,"contentLength":9},"message":"GET /bundles/kibana.style.css 200 2ms - 9.0B"}

Here's the log tail of Server2.

{"type":"log","@timestamp":"2019-05-20T16:54:28Z","tags":["status","plugin:elasti...@6.7.2","info"],"pid":1,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-05-20T16:54:28Z","tags":["info","migrations"],"pid":1,"message":"Creating index .kibana_3."}
{"type":"log","@timestamp":"2019-05-20T16:54:28Z","tags":["warning","migrations"],"pid":1,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_3 and restarting Kibana."}

[Kris Springer]

I ran 'sudo soup' on our 3rd SO Server and it's webpage now says that 'Kibana server is not ready yet' and the logs say the same message as our 2nd box about 'Another Kibana instance appears to be migrating the index. Waiting for that migration to complete'.

The good news is that Squert is still working on all boxes.

[Wes Lambert]

Hi Kris,

What is the output of the following from each server?

curl localhost:9200/_cat/indices | grep kibana

Thanks,

Wes

On Mon, May 20, 2019 at 4:29 PM Kris Springer <kspr...@innovateteam.com> wrote:

I ran 'sudo soup' on our 3rd SO Server and it's webpage now says that 'Kibana server is not ready yet' and the logs say the same message as our 2nd box about 'Another Kibana instance appears to be migrating the index. Waiting for that migration to complete'.

The good news is that Squert is still working on all boxes.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/7938e298-c47f-4243-816e-b28ca2bff39c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

https://twitter.com/therealwlambert

https://securityonion.net/

[Kris Springer]

curl outputs of each server below.

Server1
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3680 100 3680 0 0 28403 0 --:--:-- --:--:-- --:--:-- 28527
green open .kibana_2 zBlvfGBMQHihMMGNxU4dWQ 1 0 496 0 432kb 432kb
green open .kibana_1 VWqDEguKT2atqAl37vFXXA 1 0 495 146 1.7mb 1.7mb

Server2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0green open .kibana_3 zn-0GgwmRL6oWyvwiqm3Yw 1 0 492 0 448.8kb 448.8kb
green open .kibana_2 S5olWfDAS1CuuT1O2taQfQ 1 0 492 0 480.2kb 480.2kb
100 98256 100 98256 0 green open .kibana_1 yZzV3zQUSPuX-wQAsxlIqA 1 0 491 0 441.7kb 441.7kb
0 342k 0 --:--:-- --:--:-- --:--:-- 343k

Server3
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
green open .kibana ihsGuEifQLykMr5kMArvhw 1 0 487 393 2.2mb 2.2mb
100 10509 100 10509 0 0 6502 0 0:00:01 0:00:01 --:--:-- 6507
green open .kibana_2 O53DUaoDQCq8g9EideWMWQ 1 0 0 0 261b 261b

[Kris Springer]

I reran the setup on Server1 and again Squert works fine, but now Kibana webpage has a big red bar that says 'Kibana did not load properly. Check the server output for more information'

This is all very strange. I think tomorrow I'll just rebuild the server with the latest iso.

[Wes Lambert]

It may be something to do with the Kibana index migration.  We haven't noticed too many issues from our own testing, but it seems every now and then folks will run into issues.  Have you tried backing up/removing the original Kibana index after migration?  Do the logs still mention the migration is in progress?

Thanks,

Wes

On Tue, May 21, 2019 at 6:15 PM Kris Springer <kspr...@innovateteam.com> wrote:

I reran the setup on Server1 and again Squert works fine, but now Kibana webpage has a big red bar that says 'Kibana did not load properly. Check the server output for more information'

This is all very strange.  I think tomorrow I'll just rebuild the server with the latest iso.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/e4c90bda-0b1f-4b1b-b621-d13236c6027e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Dan Hoyle]

I am also having the same issue after running sudo soup on a master server.  Squert is accessible, Kibana has the red banner saying  'Kibana did not load properly. Check the server output for more information'.  Same issue with multiple browsers. 

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/CAHjBB6E0QK0118-uhAZPpNnvCM-P9m9WO-N839FGsBRjRFJ6Cg%40mail.gmail.com.

[Kris Springer]

I'll try any commands anyone suggests. At this point I'm ready to reinstall from the latest ISO, so if there's something I could try first I'm all ears.

[Dan Hoyle]

From browser inspection:

Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-SHHSeLc0bp6xt4BoVVyUy+3IbVqp3ujLaR+s+kSP5UI='), or a nonce ('nonce-...') is required to enable inline execution.

Then several errors like this:

bootstrap.js:44 GET https://10.x.x.x./built_assets/dlls/vendors.style.dll.css net::ERR_ABORTED 404 (Not Found)

then

Uncaught TypeError: Cannot read property 'dataset' of null
    at HTMLLinkElement.failure (bootstrap.js:32)

[Aurelien Guillaume]

I've the exact same error

[Wes Lambert]

What is the output of the following?

grep built_assets /etc/apache2/sites-available/securityonion.conf

Thanks,

Wes

On Thu, May 23, 2019 at 10:05 AM Aurelien Guillaume <aurelien....@itiviti.com> wrote:

I've the exact same error

--
______________________________

itiviti.com <https://www.itiviti.com/>





*The information contained in or attached to this email is strictly
confidential. If you are not the intended recipient, please notify us
immediately by telephone and return the message to us.*

*Email
communications by definition contain personal information. The Itiviti
group of companies is subject to European data protection regulations.
Itiviti’s Privacy Notice is available at www.itiviti.com
<http://www.itiviti.com>. Itiviti expects the recipient of this email to be
compliant with Itiviti’s Privacy Notice and applicable regulations. Please
advise us immediately at dataprote...@Itiviti.com if you are not
compliant with these.*

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/8ae8ef8f-552f-4370-8add-a73a3dd4971b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Wes Lambert]

It may also be that Apache has not been restarted since adding updated configuration.

You can restart it manually with:

sudo service apache2 restart

or reboot.

See the following for more details:

https://github.com/Security-Onion-Solutions/securityonion-elastic/commit/56d25bc28d4dcba2fa093e3278b1d74286fd21bf 

 Thanks,

Wes

[Dan Hoyle]

Hey Wes, for me I don't see any output for that command.  I have restarted apache and rebooted with no change.  

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/CAHjBB6FdgCoWb%3DsQj077uT7q1XwmvT8-wLAa0rvKZ6qVHU9jLQ%40mail.gmail.com.

[Wes Lambert]

It sounds like for some reason or another that file did not get updated.

What is the output of the following?

sudo dpkg -l | grep securityonion-elastic

Thanks,

Wes

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/CAA1S6bfZik4iR3BEHh08eOGm8QUF4_J2R48MWozipjDy0FHNFw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Dan Hoyle]

master$sudo dpkg -l | grep securityonion-elastic
ii  securityonion-elastic                                       20180130-1ubuntu1securityonion137          all          Elastic Stack on Security Onion

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/CAHjBB6E%2B7_YZzQ6zOyijMPaC22G9vqeH7DYdxuXdoqwRXENznw%40mail.gmail.com.

[Kris Springer]

sysadmin@SecurityOnion1:~$ sudo dpkg -l | grep securityonion-elastic
ii securityonion-elastic 20180130-1ubuntu1securityonion79 all Elastic Stack on Security Onion

[Wes Lambert]

Please try running soup again, noting any errors.

After running soup, please check the package version again.

Thanks,

Wes

On Thu, May 23, 2019 at 1:18 PM Kris Springer <kspr...@innovateteam.com> wrote:

sysadmin@SecurityOnion1:~$ sudo dpkg -l | grep securityonion-elastic
ii  securityonion-elastic  20180130-1ubuntu1securityonion79  all  Elastic Stack on Security Onion

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/310dabee-3765-463e-b75d-7483ca8947aa%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Kris Springer]

I ran soup again, no errors.

Grep output is the same as previous post.

[Wes Lambert]

Hi Kris,

Are you still on 14.04 or are you using 16.04?

Thanks,

Wes

On Thu, May 23, 2019 at 1:26 PM Kris Springer <kspr...@innovateteam.com> wrote:

I ran soup again, no errors.

Grep output is the same as previous post.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/3a2b0dc4-e371-4c5e-bdf9-047911fda83a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Kris Springer]

This box is still 14.04

Kris

You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/Y2z13igTXsU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.

To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/CAHjBB6Fnij4AQJAJX7sJZSfGV-BUDA5jLj90tAy9q212qTb2OQ%40mail.gmail.com.

[Wes Lambert]

Hi Kris,

14.04 was EOLd November 30, 2018:

https://securityonion.readthedocs.io/en/latest/eol.html 

Please use the following steps to upgrade to 16.04, and please keep in mind, the upgrade process is not 100% guaranteed.

https://securityonion.readthedocs.io/en/latest/upgrading-from-14.04-to-16.04.html?highlight=14.04

If you have further issues, please open a separate thread.  

Thanks,

Wes 

[Dan Hoyle]

Hi Wes, I am on 16.04 and having the same issue.  I do see some dependency errors

: Unmet dependencies. Try using -f.
Reading package lists...
Building dependency tree...
Reading state information...
securityonion-pfring-module is already the newest version (20121107-0ubuntu0securityonion31).
You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies:
 apt-utils : Depends: apt (= 1.2.27) but 1.2.29 is to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
E: Unmet dependencies. Try using -f.
Reading package lists...
Building dependency tree...
Reading state information...
You might want to run 'apt-get -f install' to correct these.
The following packages have unmet dependencies:
 apt-utils : Depends: apt (= 1.2.27) but 1.2.29 is installed
E: Unmet dependencies. Try using -f.
###########################################################################
All updates have been installed.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/CAHjBB6H19AdG2Nq8iSu1g6ZuHnOVfo_1nD0xdwF3YL8JUXkO3g%40mail.gmail.com.

[Wes Lambert]

Hi Dan,

Please try running the following:

apt-get -f install 

Then try running soup again 

Thanks,

Wes

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/CAA1S6bdXM_%3D_Gy9e4dxziuip4jpBMYXyfZdacj%3DD-AwOigo5Bg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Dan Hoyle]

Hi Wes,

That worked and I am now able to launch Kibana.  Thanks very much!!!!

Dan

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/CAHjBB6GJ%3DAZ6re22JTc2LxOb9t5ZodMq-EiTqe-7KYR9YkuYhA%40mail.gmail.com.

[Aurelien Guillaume]

Hi wes,
for me the output of the command you give is :

<Location /built_assets>
ProxyPass http://127.0.0.1:5601/built_assets
ProxyPassReverse http://127.0.0.1:5601/built_assets


these are 16.04 boxs with soup done without error

edit: after soup I found this in kibana logs :

{"type":"log","@timestamp":"2019-05-24T07:30:59Z","tags":["fatal","root"],"pid":1,"message":"{ [search_phase_execution_exception] all shards failed :: {\"path\":\"/.kibana/doc/_count\",\"query\":{},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"should\\\":[{\\\"bool\\\":{\\\"must\\\":[{\\\"exists\\\":{\\\"field\\\":\\\"visualization\\\"}},{\\\"bool\\\":{\\\"must_not\\\":{\\\"term\\\":{\\\"migrationVersion.visualization\\\":\\\"6.7.2\\\"}}}}]}}]}}}\",\"statusCode\":503,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[]},\\\"status\\\":503}\"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:308:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:267:7)\n at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (_stream_readable.js:1103:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)\n status: 503,\n displayName: 'ServiceUnavailable',\n message: '[search_phase_execution_exception] all shards failed',\n path: '/.kibana/doc/_count',\n query: {},\n body:\n { error:\n { root_cause: [],\n type: 'search_phase_execution_exception',\n reason: 'all shards failed',\n phase: 'query',\n grouped: true,\n failed_shards: [] },\n status: 503 },\n statusCode: 503,\n response:\n '{\"error\":{\"root_cause\":[],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[]},\"status\":503}',\n toString: [Function],\n toJSON: [Function] }"}

[Wes Lambert]

Hi Aurelien,

Have you tried restarting Apache or rebooting?

Alsop, what is the ouput of the following?

dpkg -l | grep securityonion-elastic

Thanks,

Wes

--
______________________________

itiviti.com <https://www.itiviti.com/>





*The information contained in or attached to this email is strictly
confidential. If you are not the intended recipient, please notify us
immediately by telephone and return the message to us.*

*Email
communications by definition contain personal information. The Itiviti
group of companies is subject to European data protection regulations.
Itiviti’s Privacy Notice is available at www.itiviti.com
<http://www.itiviti.com>. Itiviti expects the recipient of this email to be
compliant with Itiviti’s Privacy Notice and applicable regulations. Please
advise us immediately at dataprote...@Itiviti.com if you are not
compliant with these.*

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/b38cc287-f1b4-4939-ace3-d395eec1c317%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Aurelien Guillaume]

Hi,

yes I've rebooted multiple times,

the output of dpkg -l | grep securityonion-elastic is

ii securityonion-elastic 20190510-1ubuntu1securityonion3 all Elastic Stack on Security Onion

[Wes Lambert]

I'm not sure how you can be receiving the same error if the configuration is present.  Did you previously have the mentioned error, then run soup, then have the original issue resolved, only to experience another, different issue?

--
______________________________

itiviti.com <https://www.itiviti.com/>





*The information contained in or attached to this email is strictly
confidential. If you are not the intended recipient, please notify us
immediately by telephone and return the message to us.*

*Email
communications by definition contain personal information. The Itiviti
group of companies is subject to European data protection regulations.
Itiviti’s Privacy Notice is available at www.itiviti.com
<http://www.itiviti.com>. Itiviti expects the recipient of this email to be
compliant with Itiviti’s Privacy Notice and applicable regulations. Please
advise us immediately at dataprote...@Itiviti.com if you are not
compliant with these.*

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/965a3285-33dd-4533-9fd8-3c43381a7b7f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Aurelien Guillaume]

I was having no issue for weeks and I've been out for a week

the operation that have been done on it during this week is so-stop so-start (that can upgrade docker image),

and when I come back there was these on the servers I've tried to soup but nothing changes

[Wes Lambert]

Please provide the output of sostat-redacted for all affected boxes.

Thanks,

Wes

--
______________________________

itiviti.com <https://www.itiviti.com/>





*The information contained in or attached to this email is strictly
confidential. If you are not the intended recipient, please notify us
immediately by telephone and return the message to us.*

*Email
communications by definition contain personal information. The Itiviti
group of companies is subject to European data protection regulations.
Itiviti’s Privacy Notice is available at www.itiviti.com
<http://www.itiviti.com>. Itiviti expects the recipient of this email to be
compliant with Itiviti’s Privacy Notice and applicable regulations. Please
advise us immediately at dataprote...@Itiviti.com if you are not
compliant with these.*

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/c593b820-89ad-43f6-9d3c-4c987fe127e0%40googlegroups.com.

[Aurelien Guillaume]

here are the sostat-redacted

[Gary Lumpkin]

On Monday, May 20, 2019 at 12:02:01 PM UTC-5, Kris Springer wrote:
> I ran 'sudo soup' and after it completed and rebooted I couldn't access Kibana. I can access Squert, but not Kibana.
>
> I ran 'sudo sostat' and it says 'Logstash is running', 'Kibana is not running', 'ElastAlert is not running', 'Curator is not running'. It suggests to run 'sudo so-elastic-start', which I did. Running 'sudo sostat' again says 'Kibana is not running' but Curator and ElastAlert are now running.
>
> Running 'sudo so-elastic-start' a second time results in Kibana getting started and running successfully. But, the Kibana webpage says 'Kibana server is not ready yet'.
>
> What happened? All worked great until I ran 'sudo soup'.

I have also experienced the same errors but found that if I used the server console with Http://localhost:5601/app/kibana I could get to all my data.

[Kris Springer]

I had 4 affected SO servers. I tried all kinds of stuff, but my final solution was to re-run the setup on each. I used the opportunity to create an sosetup.conf file for each to make it easy.

I know re-running the setup may not be a good solution for everyone, but I reviewed my logs and was OK with dumping them. It cleaned everything up for me and all my SO servers are running ok now. My main server needed upgraded anyway.

I love the visibility Kibana gives! Thank you SO team!

[Aurelien Guillaume]

Hi gary, I've try you solution but I'm getting this :

please upgrade your browser

This kibana installation has strict security requirements enabled that your current browser does not meet.

any idea ?

[Wes Lambert]

As the error message suggests:

please upgrade your browser

...it may be beneficial to try upgrading your browser. 

:)

Thanks,

Wes

--
______________________________

itiviti.com <https://www.itiviti.com/>





*The information contained in or attached to this email is strictly
confidential. If you are not the intended recipient, please notify us
immediately by telephone and return the message to us.*

*Email
communications by definition contain personal information. The Itiviti
group of companies is subject to European data protection regulations.
Itiviti’s Privacy Notice is available at www.itiviti.com
<http://www.itiviti.com>. Itiviti expects the recipient of this email to be
compliant with Itiviti’s Privacy Notice and applicable regulations. Please
advise us immediately at dataprote...@Itiviti.com if you are not
compliant with these.*

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/d8168d14-e3ae-4b7c-b720-6447b527faa4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Aurelien Guillaume]

Hi, I've already do it,

i've try 3 different system: windows/windows-server/linux

on browser : chrome/firefox/egde/internet-explorer

[Fireproof Records]

ran soup this morning, broke my kibana like others are saying. Just get the red bar "kibana did not load properly" even on localhost:443. BUT, if I'm on the localhost and go to localhost:5601, kibana is fine.

so I think I have a forwarding problem, but not sure where to go to fix it.

Thoughts?

[fireproof]

ran soup this morning, now I just get "kibana did not load properly"

does not matter which machine, or what browser, any attempt to get into SERVERIP/app/kibana ends in red bar.

BUT, if I'm on the localhost and go to localhost:5601, kibana loads fine. where does the port forwarding happen in SO?

[Wes]

Fireproof,

Are you still running 14.04? If so, you'll want to upgrade to 16.04:

https://securityonion.readthedocs.io/en/latest/upgrading-from-14.04-to-16.04.html

Otherwise, did you receive any errors when running soup?

What is the result of the following?

[Aurelien Guillaume]

Hi wes,

my issue is the same as Fireproof, it's working with localhost:5601

I'm in 16.04

dpkg -l | grep securityonion-elastic give :

[Doug Burks]

Hi Aurelien,

If you're still having this issue, please start a new thread for further discussion.

Thanks!

--
______________________________

itiviti.com <https://www.itiviti.com/>





*The information contained in or attached to this email is strictly
confidential. If you are not the intended recipient, please notify us
immediately by telephone and return the message to us.*

*Email
communications by definition contain personal information. The Itiviti
group of companies is subject to European data protection regulations.
Itiviti’s Privacy Notice is available at www.itiviti.com
<http://www.itiviti.com>. Itiviti expects the recipient of this email to be
compliant with Itiviti’s Privacy Notice and applicable regulations. Please
advise us immediately at dataprote...@Itiviti.com if you are not
compliant with these.*

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/f462e675-6722-489f-af5b-69f8317d62ea%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Doug Burks
CEO
Security Onion Solutions, LLC

[Aurelien Guillaume]

Hi doug,
there is already one : https://groups.google.com/forum/#!topic/security-onion/ydCtPTgR_K8

Thanks

[Brian Green]

Hi,  I have the same problem after I ran soup; everything ran fine but Kibana shows error "Kibana did not load properly. Check the server output for more information"

I end up deleting and recreating  the index but still I have the same issue and there is no error in the kibana logs

I'm running SO 16.04 and when I reran soup everything showed it was up to date.

Some output:

 dpkg -l | grep securityonion-elastic

ii  securityonion-elastic                                       20180130-1ubuntu1securityonion153          all          Elastic Stack on Security Onion

 

curl -X GET "localhost:9200/_cat/indices/.kib*?v&s=index"

health status index     uuid                   pri rep docs.count docs.deleted store.size pri.store.size

green  open   .kibana_1 dRHUM7jhQuu1SnT8LZyh0w   1   0        490          144      1.7mb          1.7mb

Any help is much appreciated.

Regards,

Brian

On Monday, May 20, 2019 at 1:39:09 PM UTC-4, Doug Burks wrote:

Hi Kris,

Have you checked the Kibana log for any additional clues?

/var/log/kibana/kibana.log

On Mon, May 20, 2019 at 1:33 PM Kris Springer <kspr...@innovateteam.com> wrote:

I ran 'sudo soup' on a second SO Server of ours and the exact same issues have occurred.  Kibana is not working on a second server now.

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to securit...@googlegroups.com.

To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/efc89da7-89dd-4a5c-aae7-3eaa352038f9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Kris Springer]

My Kibana breaks every time I run soup.  I've kept notes on the commands to try to make things work again after soup and reboot.    

sudo docker start so-kibana

sudo so-elastic-start

sudo so-elastic-configure-kibana

Then refresh your browser to re-login to Kibana.

Hope this helps you.

[Doug Burks]

Hi Kris,

We're currently working on Elastic 6.8.4 and we have a change which might help with this issue in the future:

https://github.com/Security-Onion-Solutions/security-onion/issues/1655

Once our Elastic 6.8.4 integration is ready for testing, it'd be great if you could test and see whether or not it helps.

--

Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/67ee827f-eacc-4045-a6dd-b924aa003e7b%40googlegroups.com.

[Eric Osterberg]

I have seen the same issue for about the last 6 weeks. To resolve I simply run sudo so-kibana-start

New in the last 3 to 8 days, I am also finding the salt-master is not running after a master reboot. Fixed with sudo systemctl start salt-master

[Doug Burks]

Hi Eric,

I don't think your salt-master issue is related to Kibana.  If you would like to troubleshoot your salt-master issue, please start a separate thread and include more detailed information from the log.

--

Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/73a5e796-e68d-4760-b206-387c4599331c%40googlegroups.com.

[Doug Burks]

Hi Kris,

As a follow up to my comment below, we've just released Elastic 6.8.4 for testing:

https://groups.google.com/d/topic/security-onion-testing/UwpKpKbdBEo/discussion

If you have a test environment that you can test this in before we do the final release, that would be great.    Please let us know whether or not it helps.

Thanks!

[Doug Burks]

Hi Kris,

As a follow up to my follow up below, have you had a chance to test Elastic 6.8.4?  We are hoping to release next week.

Thanks!

[Kris Springer]

I upgraded to v6.8.4 and it seems to be working fine with no issues.

Kris Springer

You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/Y2z13igTXsU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/CAJ%2BhwWCxVYUC6Rb9K32bJb%2BfjYAc%2B20OdFbHjjjAYcdqkAB-kw%40mail.gmail.com.

[Doug Burks]

Excellent, thanks Kris!