index pattern and template not found

[rapiertg]

Hello,

Since migration to 4.x I am struggling with below error. Cannot access Wazuh as it does not go thru health check. I was able make it work quite randomly on some versions after fighting with it a lot and never knew what was the reason. This is migrated from pre 4.x.

I set it up  manually using all in one documentation with elasticsearch basic licence. The weird thing is that template and pattern is there (created by filebeat) - for some reason wazuh app just doesn't see it.

Also I have some weird browser side errors:

wazuh.plugin.js:2 POST https://server/api/saved_objects/index-pattern/wazuh-alerts-* 409 (Conflict)

Error: Minified React error #31; visit https://reactjs.org/docs/error-decoder.html?invariant=31&args[]=TypeError%3A%20Cannot%20read%20property%20'id'%20of%20undefined&args[]= for the full message or use the non-minified dev environment for full errors and additional helpful warnings.

GET https://server/api/saved_objects/index-pattern 404 (Not Found)

Error: Minified React error #31; visit https://reactjs.org/docs/error-decoder.html?invariant=31&args[]=TypeError%3A%20Cannot%20read%20property%20'id'%20of%20undefined&args[]= for the full message or use the non-minified dev environment for full errors and additional helpful warnings.

GET https://server/api/saved_objects/index-pattern 404 (Not Found)

Kibana is revere proxied with nginx, but after disabling it and using kibana alone it is completelly the same.

cat /var/log/filebeat/filebeat | grep -i -E "error|warn" and cat /var/log/elasticsearch/elasticsearch.log | grep -i -E "error|warn" are empty

Will be very grateful for helping with this.

[Caio Oliveira]

Hi Rapiertg,

I'm having the same problem, except that "template" is fine.If you fix that, share with me. Every day I spend some hours trying to fix that.

Thanks

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/fddbaffa-51bc-4dc2-9ac4-1bc947fddf48n%40googlegroups.com.

[Franco Charriol]

Hi guys,
Are you using OpenDistro or X-Pack in addition to Wazuh?
In that case, please ensure that the user set as 'elasticsearch.user' in '/etc/kibana/kibana.yml' has all permission over 'wazuh*' indices.

If this is not your problem maybe are you using custom index patterns or custom indices prefix?

Regards
Franco

[Caio Oliveira]

Hi,

I’m using XPACk with security enable. On kibana.yml I set the user “elastic” that is a superuser.

I tried to configure custom index pattern and pattern id with wazuh-alerts-* and tried to let the Wazuh App create the index pattern.

None of this options fix the problem.

Ah, the index exist and the events are ok, I can see everything in Discovery. Just the wazuh app has problem to open.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/8769077e-4b15-4ac2-804f-a9bc8871c831n%40googlegroups.com.

[rapiertg]

Hi,

I am also using X-Pack. It was elastic user before. Just to test it out I just created new user with superuser and kibana_admin roles - it is the same.

I never tried to change index, was always using the default.

Regards

[Franco Charriol]

oh, I see.
What version and revision of Wazuh and Elastic are you using?
For the Wazuh app, you could check it in the section Settings / About.

[rapiertg]

I cannot access this tab as I am not getting thru self check, but I am sure I am on:
App: 4.1.2

ELK:  7.10.2

Have this issue since Wazuh 4.x. Sometimes it eventually started to work on some versions without any reason until next update.

[Caio Oliveira]

I'm using the same version:

App: 4.1.2

ELK:  7.10.2

I have another ELK+Wazuh with Kibana app 4.0.x and ELK 7.9.x and everything is fine.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/7cc29ea5-3b64-4561-b45b-033f6fc9eb2cn%40googlegroups.com.

[Franco Charriol]

can you share the list of saving objects of Elastic, please?

you can find it in Kibana (menu) / Stack Management / Saved Objects

we found a problem with the size of the payload of some objects because the default value of this Elastic setting is 1MB.

so when you try to see this section of Kibana it throws, otherwise if this is not the case we can check if you have an old pattern or something else.

You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/H0yYslTaxEo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/CAEDh8dscRxY4M-uKGh2w2xGi8q_r5BcNHd_%3DLUT92yrvBHpKSA%40mail.gmail.com.

[Caio Oliveira]

No problem to access this, except by the high memory use.

I this picture I click on "relationships"

[Franco Charriol]

I see, cloud you clear your browser cache o try in incognito mode?

Sometimes the index pattern is kept in the cookies of the page.

[rapiertg]

I tried clearing it multiple times, also multiple browsers. No luck. My saved objects look like in attached image (filtered with wazuh* as I have more then 800 saved ones from other apps):

[Franco Charriol]

do you see any error message in the browser console when the Health-check runs?

Also, you could check the app logs in `/usr/share/kibana/data/wazuh/logs/wazuhapp.log` to find some relevant error

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/02ec22e9-8c75-4ccc-8775-f30f3c72d35an%40googlegroups.com.

[rapiertg]

Yes, as mentioned in initial message post I get those errors in browser:

wazuh.plugin.js:2 POST https://xxxx/api/saved_objects/index-pattern/wazuh-alerts-* 409 (Conflict)

Error: Minified React error #31; visit https://reactjs.org/docs/error-decoder.html?invariant=31&args[]=TypeError%3A%20Cannot%20read%20property%20'id'%20of%20undefined&args[]= for the full message or use the non-minified dev environment for full errors and additional helpful warnings.

GET https://xxxx/api/saved_objects/index-pattern 404 (Not Found)

Error: Minified React error #31; visit https://reactjs.org/docs/error-decoder.html?invariant=31&args[]=TypeError%3A%20Cannot%20read%20property%20'id'%20of%20undefined&args[]= for the full message or use the non-minified dev environment for full errors and additional helpful warnings.

GET https://xxxx/api/saved_objects/index-pattern 404 (Not Found)

But no errors in /usr/share/kibana/data/wazuh/logs/wazuhapp.log.

If i delete pattern manually and open the app, the new one is created but same errors occurs.

[Franco Charriol]

you could force to overwrite the index pattern with the elastic API

you should follow these steps:

1.  Open Kibana web press F12 and select the tab Network (filter by XHR request)
   
2. Login and check any of the requests in the table and in the sub-tab Headers the value of the header Cookie of the section Request Headers and keep it
   
   
   
3. Go to Kibana / Saved Objects and remove the index pattern wazuh-alerts-*
4. Now open a console or Postman and try to run this request
   

   curl -X POST 'https://<KIBANA-URL>/api/saved_objects/index-pattern/wazuh-alerts-*?overwrite=true' -H "Content-Type: application/json"  -H 'kbn-xsrf: kibana' -H 'Cookie: <YOUR-COOKIE-VALUE>' -d '{ "attributes": {
           "title" : "wazuh-alerts-*",
           "id" : "wazuh-alerts-*"
   }}'
   

5. Clear all cache browser by right click  on the refresh button and select Empty cache and hard reload
   
6. Open the Wazuh App
   
   
   

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/b0f6d112-93a1-4e7c-801b-c6ffe15ccba6n%40googlegroups.com.

[rapiertg]

I followed your instructions. After forcing index overwrite I get this as an output:
{"type":"index-pattern","id":"wazuh-alerts-*","attributes":{"title":"wazuh-alerts-*","id":"wazuh-alerts-*"},"references":[],"migrationVersion":{"index-pattern":"7.6.0"},"updated_at":"2021-03-11T21:25:02.436Z","version":"WzI0NDcsM10=","namespaces":["default"]}

But App is still not working.

[Franco Charriol]

A moment ago a user with a similar problem told me that the only way he found to solve it was removing from Elastic API the '.kibana' index with Kibana service down.

Deleted the .kibana index, restarted kibana, kibana complained that there was a migrating index and it couldn't start. Deleted the .kibana index from ES directly while kibana was off, then started kibana again. Did a hard refresh of the tab, and wazuh app loaded!

But be careful with these the .kibana index has all saved objects, so you need first export your all saved objects (except Wazuh index patterns) from Kibana / Stack Management / Saved Objects

I suggest exporting them in little chunks (little ndjson files) to avoid the 'too big file' error from Kibana when you try to re-import the objects.

An alternative cloud be reindexing the .kibana index with this guide as example https://www.elastic.co/guide/en/kibana/6.8/migrating-6.0-index.html

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/7b907dc8-444e-4470-b6e3-8f550666a754n%40googlegroups.com.

[Caio Oliveira]

Franco,

Thanks for your help. 

Delete the .kibana worked for me.

But I didn't pay attention to "too big file"problem and now I got this problem. 

Hahahaha 

Now I will research how to fix that.

Thanks

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/CAMuL%3Dvxa5FEg5_Z7yjkh21wfVvwdH-CUc36x1-TzEgxxw%2BkgJQ%40mail.gmail.com.

[rapiertg]

Thank you, that helped! Would never think of deleting kibana index would help in this case. After 3 months of fight wazuh is working again :). Appreciate.

@Caio I just chopped the ndjson file into smaller chunks manually after export with about 200 lines each file, which got rid of error.

[Franco Charriol]

I'm glad I could help you.
If you have any further questions, please do not hesitate to ask. Also, you can use our slack channel (https://wazuh.com/community/join-us-on-slack/).
Kind Regards

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/0f130c05-2548-4a5e-92cf-7b83c6eb27e8n%40googlegroups.com.

[Caio Oliveira]

I fixed the import of the ndjon. Everything is fine.Thanks for all

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/CAMuL%3DvxymXQQBmtX7aFwi%3DCXAL6uSc9LvV5%3Df6LVjCHnS270qQ%40mail.gmail.com.

[Franco Charriol]

great! 💪

[Vishakh L]

Doesn't deleting .kibana indices cause failure in loading kibana console?