EHOSTUNREACH after upgrade Elasticsearch and Kibana

[Oriol Val Gangonells]

Hello everyone.

I have recently updated Elasticsearch version and Kibana version from 5.6.4 to 6.2.4. Mainly it's all correct.

I update Wazuh to 3.2.1 versión too.

Then, when I try to turn on Kibana doing: /usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml or service kibana start, I get this error:

  log   [09:20:03.747] [error][wazuh][monitoring] connect EHOSTUNREACH xxx.xxx.xxx.xxxx:yyy

Where, xxx.xxx.xxx.xxx is the OSSEC Server IP and yyy is the Kibana port.

I would be grateful if someone answers me.

Thanks.

Oriol

[jesus.g...@wazuh.com]

Hello Oriol,

from your message I see your monitoring module is having troubles connecting at some point.

The monitoring module is simply a cron task which make requests to the Wazuh API and insert data on the

wazuh-monitoring-* indices on Elasticsearch. 

The monitoring module make requests pointing to the Wazuh API address and the Wazuh API port (usually 55000).

Once the data is fetched it inserts data on Elasticsearch (usually on port 9200). 

Your Wazuh API data should be stored at .wazuh index, please paste the output of the next curl command

(remove credentials once you are going to paste the content here):

# curl elastic_ip:9200/.wazuh/_search?pretty

The above command gives us useful information about your stored configuration.

When you say OSSEC Server IP I'm assuming Wazuh manager IP, right? if so, please paste the ouput of this curl command:

# curl ossec_ip:55000 -u apiuser:apipass

This way we can check if your Wazuh API is down or not.

It would be nice if you give us the Wazuh App logs content, you could do it as follow:

# cat /usr/share/kibana/plugins/wazuh-logs/wazuhapp.log

With the logs we can check certain messages to ensure what's happening.

Finally, let us know your App revision code as follow:

# cat /usr/share/kibana/plugins/wazuh/package.json | grep revision

Maybe your revision is lower than 389 since we released it just yesterday, I suggest you to upgrade it

and we can help you to upgrade it on a easy way.

That's all for now, regards,

Jesús

[Oriol Val Gangonells]

Thank you Jesús.

In fact, something that I have not said is that the IP that is giving me the error is wrong. I had this IP but I changed it, and then I have done this procedure of ugrade Elastic, Kibana and Wazuh.

Here is the output of the first command curl elastic_ip:9200/.wazuh/_search?pretty. In "url" field, appears the same IP that gives me the error of EHOSTUNREACH, that is wrong. Do you know how to change it?

 

{

  "took" : 13,

  "timed_out" : false,

  "_shards" : {

    "total" : 5,

    "successful" : 5,

    "skipped" : 0,

    "failed" : 0

  },

  "hits" : {

    "total" : 1,

    "max_score" : 1.0,

    "hits" : [

      {

        "_index" : ".wazuh",

        "_type" : "wazuh-configuration",

        "_id" : "zzzzz",

        "_score" : 1.0,

        "_source" : {

          "component" : "API",

          "extensions" : {

            "audit" : true,

            "pci" : true,

            "oscap" : true

          },

          "manager" : "server_OSSEC_name",

          "insecure" : "true",

          "active" : "true",

          "api_user" : "R3",

          "api_password" : "xxxx",

          "url" : "@ip_server_OSSEC_name", -----------> This IP is wrong, i have to change it

          "api_port" : "55000",

          "cluster_info" : {

            "node" : "nodata",

            "cluster" : "nodata",

            "manager" : "server_OSSEC_name",

            "status" : "disabled"

          }

        }

      }

    ]

  }

}

Thank you so much.

Oriol

El dijous, 26 abril de 2018 12:01:42 UTC+2, jesus.g...@wazuh.com va escriure:

[jesus.g...@wazuh.com]

Sure Oriol,

the query you are looking for is as follow:

# curl -XPOST elastic_ip:9200/.wazuh/wazuh-configuration/1524561570158/_update -H 'Content-Type: application/json' -d'{"doc":{"url":"http://192.168.1.62"}}'

Where http could be https if you are using it, and 192.168.1.62 should be your new IP. This number, 1524561570158, is exactly the field _id which you said "_id" : "zzzzz".

Hope it helps Oriol. Once finished, you should see something like this:

{"_index":".wazuh","_type":"wazuh-configuration","_id":"1524561570158","_version":4,"result":"updated","_shards":{"total":2,"successful":2,"failed":0},"_seq_no":5,"_primary_term":30}

Depending on your Elasticsearch architecture it could be different.

Restart Kibana and wait some seconds before use it:

# systemctl restart kibana

If all went fine the error should disappear. Also remember you can do few checks looking at the wazuhapp.log file.

If you have any trouble doing this we can help you, but for sure the error was the wrong IP.

Let us know once done!

Best regards,

Jesús

[Oriol Val Gangonells]

Jesús,

All was good. Thanks.

But now, the error is which follows, when I try to start kibana:

  log   [10:54:08.258] [error][wazuh][monitoring] socket hang up

I don't know if it's for the same reason.

Best regards.

Oriol

[jesus.g...@wazuh.com]

Ok Oriol it seems like a protocol error maybe or something wrong connecting to the Wazuh API.

95% times I found this error connecting to the Wazuh API it was for the HTTP vs HTTPS usage.

If your Wazuh API is http://10.0.0.1 with port 55000 and user 'foo' and password 'bar' check this

from the server where the Wazuh App is installed along Kibana:

# curl http://10.0.0.1:55000 -u foo:bar

This way we can check if indeed that server can connect with the API server. Any case all is fine the only problem here

is connectivity so don't worry we can solve it. Also the above command should gives you the Wazuh API version which is running (should be 3.2.1).

Once you check which is your right protocol, port, etc. repeat the curl commands

provided to get the .wazuh index and to update it properly.

Additionally: check the Wazuh API status con the manager server using this:

# systemctl status wazuh-api

Best regards,

Jesús

[Oriol Val Gangonells]

Ok, what was you said.

Sorry to question so much, but now I can connect to Kibana and when I go to wazuh and I try to edit the api, it shows the next message:

Settings. The request to /api/wazuh-api/checkAPI took too long and was aborted

I have changed the credentials in wazuh, and try to insert the new ones, but doesn't work.

If I do service wazuh-api status it gives me active.

Thanks in advance.

Oriol

 Settings. The request to /api/wazuh-api/checkAPI took too long and was aborted

[jesus.g...@wazuh.com]

Ok Oriol, 

let's leave empty your .wazuh index (as I said, it only stores API credentials) as follow:

# curl -XPOST 'elastic_ip:9200/.wazuh/wazuh-configuration/_delete_by_query?conflicts=proceed&pretty' -d'{"query":{"match_all":{}}}' -H 'Content-Type: application/json'

Now restart Kibana and wait few seconds:

# systemctl restart kibana

Once it's finished, open a new incognito window(to discard any cache, cookies problem because I don't know your App revision) on your browser and enter Kibana.

It'll ask you to insert your Wazuh API credentials, use the form to insert them. Before insert any data, open the developers tools of your browser and look at the Network

tab, then fill the form and click on Save API, if an error occurred but the app is not so clear, the Network tab will give us the key here.

Regards,

Jesús

[Oriol Val Gangonells]

Ok Jesús, sorry for the delay. 

 

I have left empty my .wazuh index, and the output was the following:

 {

  "took" : 140,

  "timed_out" : false,

  "total" : 1,

  "deleted" : 1,

  "batches" : 1,

  "version_conflicts" : 0,

  "noops" : 0,

  "retries" : {

    "bulk" : 0,

    "search" : 0

  },

  "throttled_millis" : 0,

  "requests_per_second" : -1.0,

  "throttled_until_millis" : 0,

  "failures" : [ ]

}

After restart Kibana, I open an incognito window and I go to Dev tools, but I don't see the Network Tab. When I enter my Wazuh credentials I got the same error.

My revision App is 389 like you said before.

Best regards.

Oriol

[Oriol Val Gangonells]

Sorry Jesus, I think I have found what you referred. 

I think that the network tab you referred is clicking f12.

What I obtained is the following:

1. {statusCode: 500, error: 5, message: "socket hang up"}
1. error:5
2. message:"socket hang up"
3. statusCode:500

[jesus.g...@wazuh.com]

Hello Oriol, 

the problem is most probably due to a wrong protocol. You are using https pointing to http or you are using http pointing to https.

My suggestion is to give a try to the next command:

# curl wazuh_api_url:wazuh_api_port -u wazuh_api_user:wazuh_api_pass

Example: curl http://10.0.0.2:55000 -u foo:bar

Try the above command from the same machine where Kibana is installed. This way we can check if your Kibana machine can reach the Wazuh API

host. Hope it helps.

Also it would be nice if you paste the full information related to the error message you gave us. It means you could paste the URL of the failed request,

and any information related to that request. You can check it at the same tab from the Dev Tools.

Best regards,

Jesús

[jesus.g...@wazuh.com]

Yes, I've seen you have deleted it, now I can confirm you that your message is no longer avalaible.

El viernes, 27 de abril de 2018, 11:35:43 (UTC+2), Oriol escribió:

Oh, I didn't show them... Please, can you delete your message due to my privacy?

Thanks.

[jesus.g...@wazuh.com]

Ok, check it again, I think now there is no text about your last message.

El viernes, 27 de abril de 2018, 11:45:33 (UTC+2), Oriol escribió:

In your before message, i can see my message. Please, delete your message to avoid seeing.

Thanks

[Oriol]

It's perfect.

Thank you so much Jesús. And I'm sorry for the inconveniences.

Oriol

[Jesús Ángel González]

You are welcome Oriol, no problem!

Best regards,

Jesús

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/dd1957dd-7baf-4016-a2b5-c247e6a00ad0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Jesús Ángel González Novez

jesus.g...@wazuh.com

https://wazuh.com