How to delete unwanted documents from index files

[ismailctest C]

Hi Team,

Index file size is too large, please let us know how to delete unwanted documents from old index files.

Note:

We have applied rules and needed logs only forwarding to ELK.

Needed index files for future reference because of we can't remove all index files. So, need to delete only selected fields/docs from index files. How to do?

[Jesus Linares]

Hello,

Well, you have several options to safe space in your indexer:

1. Configure properly your decoders/rules. Every event that matched a rule with a level higher or equal to 3 is indexed. So, review your rule to ignore the events that you don't want to index. Also, in the decoder you can decide the fields to extract.
2. Also, in your filebeat/logstash configuration, you can remove the fields that you don't need before indexing them. Example: https://www.elastic.co/guide/en/beats/filebeat/current/drop-fields.html.
3. Once the alert is indexed, try to keep only the indices that you need. For example, the last 3 months. You can do it manually with the API or using the Index State Management. Also, you could create a snapshot in a repository like S3 (see more information).

> need to delete only selected fields/docs from index files. How to do?

You can use "delete by query" to delete all documents that match a query: https://opensearch.org/docs/1.0/opensearch/rest-api/document-apis/delete-by-query/

I hope it helps.

Regards.