Too many "410 Gone encountered. Restarting pod watch to reset resource versions" log message in fluentd

[chaitra hegde]

Hi,

  I am using td-agent rpm 3.8.0 which contains fluentd 1.11.1 gem. I have installed fluent-plugin-kubernetes_metadata_filter (2.5.2) version. I am using k8s environment and reading container logs. I can see the below log message repeatedly. There are no other error logs in fluentd and fluentd pods are running fine.

I have set  K8S_NODE_NAME env variable in fluentd pods( https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter ) because fluentd process was consuming too much memory and pods were getting restarted due to OOMKilled error frequently. Are the below logs due to this env variable setting? Is there any problem with fluentd having below log messages? Also I am curious to know why is this log message repeatedly appearing in fluentd logs. Any idea on this?

2021-05-26 03:41:06 +0000 [info]: #0 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone

2021-05-26 04:05:17 +0000 [info]: 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone

2021-05-26 04:34:37 +0000 [info]: #0 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone

2021-05-26 05:02:00 +0000 [info]: 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone

2021-05-26 05:05:16 +0000 [info]: #0 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone

2021-05-26 05:39:49 +0000 [info]: 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone

2021-05-26 06:00:29 +0000 [info]: #0 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone

2021-05-26 06:15:44 +0000 [info]: 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone

2021-05-26 06:51:55 +0000 [info]: #0 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone

2021-05-26 07:10:13 +0000 [info]: 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone

2021-05-26 07:44:40 +0000 [info]: #0 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone

2021-05-26 08:02:10 +0000 [info]: 410 Gone encountered. Restarting pod watch to reset resource versions.410 Gone