Missing Index Exception

[Rhys Campbell]

Hi All,

I get the following error on my primary fluentd instance today (date 2014-08-18)...

2014-08-18 16:03:21 +0100 [warn]: temporarily failed to flush the buffer. next_retry=2014-08-18 16:07:18 +0100 error_class="Elasticsearch::Transport::Transport::Errors::NotFound" error="[404] {\"error\":\"IndexMissingException[[logstash-2014.08.07] missing]\",\"status\":404}" instance=70340969446260

This causes all updates, from all nodes to cease until I restart the td-agent service. fluentd seems to want to publish to an index that does not exist. I thought this might be due to one of my fluentd nodes being set to an incorrect date but this is not the case. 

A restart resolves the issue but it comes back after 30 minutes or so. Any ideas what is going on here?

Cheers,

Rhys

[Christian Hedegaard]

The elasticsearch plugin should just automatically create/re-create the index if it’s missing. Something else is happening here.

 

I had a similar issue, and from what I can recall it was that the index was there but the fields/mapping was incorrect so it couldn’t put the information into the index. I believe the error was the same.

 

If ES is throwing a 404 you should look at the logs on all your nodes to find out which node is returning the 404 and why. It could also be a bad node that needs to be restarted or replaced.

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rhys Campbell]

Hi Christian,

Apologies for the late response. I've been away from work. My EFK system is just a demo system running on a single node.

I do run a maintenance script that closes and then archives indices older than X days. I think I am doing this correctly. I use the elasticsearch head plugin and this only lists the indices I expect to be there and does not mention the ones I get in the td-agent log file.

I have set "ignore_unavailable: true" in my elasticsearch config file. I know this is only really masking the problem but since it's only a demo system I've gone with it for now. 

Cheers,

Rhys

[Rhys Campbell]

Just a quick update on this...

While the es config change mentioned above seemed to resolve the issue it did occasionally reoccur. The old indexes were sometimes re-create, while on other occasions the same MissingIndexEception was thrown.

The old indexes that were created only ever contained 1-5 records and these were always from the same host. The date settings of this server were fine but I did notice that the syslog contained out-of-order messages. i.e.

2014-09-04T18:00:02.001593+01:00

2014-09-04T18:00:02.001848+01:00

2014-08-12T06:30:01.092147+01:00

2014-09-04T18:00:02.002107+01:00

2014-09-04T18:00:02.017149+01:00 

No sure why this would be happening but for now I'm going to stop this box sending the syslog to EFK.

Cheers,

Rhys