Observability-Event Analytics-Explorer: Error index not found

[María A]

Hi, super team

I am testing the last Wazuh version. I need to search data with Event Analytics, but is not showing up any result because an error:

source = `wazuh-alerts-4.x-2022.05.23` 

archives.json:
{"timestamp":"2022-05-23T10:54:56.055+0200","rule":{"level":2,"description":"Unknown problem somewhere in the system.","id":"1002","firedtimes":139,"mail":false,"groups":["syslog","errors"],"gpg13":["4.3"]},"agent":{"id":"000","name":"siem"},"manager":{"name":"siem"},"id":"1653296096.545321","full_log":"May 23 10:54:55 siem opensearch-dashboards[625]: Event analytics fetch error:  { Error: [status_exception] ObservabilityObject `wazuh-alerts-4.x-2022.05.20` not found","predecoder":{"program_name":"opensearch-dashboards","timestamp":"May 23 10:54:55","hostname":"siem"},"decoder":{},"location":"/var/log/syslog"}

 

I have read in OpenSearch documentation that back ticks are needed when index name has some symbols in it, but is not working either with source = `wazuh-alerts-*` 

I don't know if I need to do something with indexes before search with Event Analytics.

Thank you so much!

[Antonio David Gutiérrez]

Hi @Maria A

Thank you for using Wazuh.

I go to try to replicate your problem. I guess you are using a Wazuh dashboard and Wazuh indexer. What Wazuh version are you using?

Could you share your use case?

[Antonio David Gutiérrez]

Hi María A,

I was researching your problem and I could replicate it.

Using the Wazuh dashboard for Wazuh 4.3.0, that uses as base an OpenSearch Dashboards 1.2.0.

When you do a search in `Observability/Event analytics`, there are 3 requests:

R1. Check if exists a saved query.

Endpoint: <WAZUH_DASHBOARD_ADDRESS>/api/observability/event_analytics/saved_objects?objectId=<SOURCE_INDEX>
where:
  - <WAZUH_DASHBOARD_ADDRESS> is the adddres of the Wazuh dashboard including the port
  - <SOURCE_INDEX> is the index where searching

In my case, the response contains a similar error to what you posted: Error: [status_exception] ObservabilityObject `wazuh-alerts-4.x-2022.05.20` not found independently of index name (or using the identifier delimiters or not). I think it is related that doesn't exist a saved query that match. I don't know the purpose of this request or if the result could affecting to the next steps.

R2. Get the index field mappings.

Endpoint: <WAZUH_DASHBOARD_ADDRESS>/api/dsl/indices.getFieldMapping?index=<SOURCE_INDEX> 
where:
  - <WAZUH_DASHBOARD_ADDRESS> is the adddres of the Wazuh dashboard including the port
  - <SOURCE_INDEX> is the index where searching

R3. Search using the Piped Processing Language (PPL) query.

Endpoint: <WAZUH_DASHBOARD_ADDRESS>/api/ppl/search
Endpoint JSON payload: {format, query}

Cases:

C1. Without backticks as identifier delimiter for the index name
Search: source = wazuh-alerts-4.x-2022.05.25 | fields agent

R1: The response contains a similar error you posted.
R2: Get the index fields mapping
  Request: <WAZUH_DASHBOARD_ADDRESS>/api/dsl/indices.getFieldMapping?index=wazuh-alerts-4.x-2022.05.25
  Note that the backtick characters were URL encoded.
R3: The search fails and the response is:
'{\n  "error": {\n    "reason": "Invalid Query",\n    "details": "Failed to parse query due to offending symbol [x-2022.05.25] at: \'source=wazuh-alerts-4.x-2022.05.25\' <--- HERE... More details: Expecting tokens in {\'SEARCH\', \'SOURCE\', \'INDEX\', \'D\', \'NOT\', \'TRUE\', \'FALSE\', \'INTERVAL\', \'MICROSECOND\', \'MILLISECOND\', \'SECOND\', \'MINUTE\', \'HOUR\', \'DAY\', \'WEEK\', \'MONTH\', \'QUARTER\', \'YEAR\', \'.\', \'+\', \'-\', \'(\', \'`\', \'AVG\', \'COUNT\', \'MAX\', \'MIN\', \'SUM\', \'VAR_SAMP\', \'VAR_POP\', \'STDDEV_SAMP\', \'STDDEV_POP\', \'FIRST\', \'LAST\', \'ABS\', \'CEIL\', \'CEILING\', \'CONV\', \'CRC32\', \'E\', \'EXP\', \'FLOOR\', \'LN\', \'LOG\', \'LOG10\', \'LOG2\', \'MOD\', \'PI\', \'POW\', \'POWER\', \'RAND\', \'ROUND\', \'SIGN\', \'SQRT\', \'TRUNCATE\', \'ACOS\', \'ASIN\', \'ATAN\', \'ATAN2\', \'COS\', \'COT\', \'DEGREES\', \'RADIANS\', \'SIN\', \'TAN\', \'ADDDATE\', \'DATE\', \'DATE_ADD\', \'DATE_SUB\', \'DAYOFMONTH\', \'DAYOFWEEK\', \'DAYOFYEAR\', \'DAYNAME\', \'FROM_DAYS\', \'MONTHNAME\', \'SUBDATE\', \'TIME\', \'TIME_TO_SEC\', \'TIMESTAMP\', \'DATE_FORMAT\', \'TO_DAYS\', \'SUBSTR\', \'SUBSTRING\', \'LTRIM\', \'RTRIM\', \'TRIM\', \'LOWER\', \'UPPER\', \'CONCAT\', \'CONCAT_WS\', \'LENGTH\', \'STRCMP\', \'RIGHT\', \'LEFT\', \'ASCII\', \'LOCATE\', \'REPLACE\', \'LIKE\', \'ISNULL\', \'ISNOTNULL\', \'IFNULL\', \'NULLIF\', \'IF\', \'MATCH\', \'SPAN\', \'MS\', \'S\', \'M\', \'H\', \'W\', \'Q\', \'Y\', ID, INTEGER_LITERAL, DECIMAL_LITERAL, DQUOTA_STRING, SQUOTA_STRING, BQUOTA_STRING}",\n    "type": "SyntaxCheckException"\n  },\n  "status": 400\n}'

When the dot character ( .) is found in the index name, there is an error in the query.

C2. With backticks as identifier delimiter for the index name
Search: source = `wazuh-alerts-4.x-2022.05.25` | fields agent

R1: The response contains a similar error you posted.
R2: Fails
  Request: <WAZUH_DASHBOARD_ADDRESS>/api/dsl/indices.getFieldMapping?index=%60wazuh-alerts-4.x-2022.05.25%60
  Note that the backtick characters of the index name were URL encoded.
R3: Not done

I tested using a request to the API endpoint of ppl as explained here and it works using the backticks as identifier delimiter.

I tested using OpenSearch Dashboards 1.3.0 and I got the same problem. I reviewed the release notes for 1.3.0 and 2.0.0 and at the moment, I didn't see any fix related to this problem. The OpenSearch team is working in 2.0.0 so maybe is not finished and could include more changes.

My conclusion is that there is some bug in the Observability plugin that doesn't manage/parse/do the requests correctly. This plugin is developed by the OpenSearch team, so you could be interested to open an issue in the GitHub repository explaining the case.

As a workaround, you could use an index whose name doesn't contain special characters as explained in OpenSearch regular identifiers documentation. To do this, you will need:
1. Create a target index whose name doesn't contain the special characters defining the mappings. You could be interested to extend the wazuh template used by the wazuh-alerts-4.x-* and wazuh-archives-4.x-* indices to include the new index you will create in the next step.
2. Reindex the data. More information: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/docs-reindex.html#docs-reindex.
3. Use the new index to search in Observability/Event analytics

[María A]

Thank you so much, Antonio (and sorry everyone because I did not realize that I replied to the email only to Antonio with use case data.) I opened an issue in OpenSearch, as you suggested.

I tried to create another index to change the name and reindex data but is not working properly. Where can I see the process to create another index and reindex? I think that I am doing something wrong.

Thank you!