failed to execute bulk item (index) index on EFK logging stack

2,040 views
Skip to first unread message

zhuzhen...@gmail.com

unread,
Dec 8, 2016, 3:37:01 AM12/8/16
to Fluentd Google Group
Hi, groups,
    I am using EFK logging stack on kubernetes cluster. If I don't comment out the following 3 lines in td-agent.conf:  
<filter kubernetes.**>
  type kubernetes_metadata
</filter> 

I'll get ES warnning like this (my  td-agent version is 0.12.29 ):

[2016-12-08 07:16:56,802][DEBUG][action.bulk              ] [Eleggua] [logstash-2016.12.08][3] failed to execute bulk item (index) index {[logstash-2016.12.08][fluentd][AVjdSsJc-DLL_mWZ7vgs], source[{"log":"E1208 15:02:42.196089       1 reflector.go:214] k8s.io/contrib/ingress-admin/loadbalancer-controller/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:109: Failed to list *runtime.Unstructured: the server could not find the requested resource\n","stream":"stderr","docker":{"container_id":"02fa6a33b7feb47e7508bd2dfea8fe47a9edfc4ce46a8539e1f7219ad44bea51"},"kubernetes":{"namespace_name":"kube-system","pod_id":"7a95f891-bab9-11e6-865a-0800274a55b6","pod_name":"loadbalancer-controller-wn9ws","container_name":"loadbalancer-claim-controller","labels":{"run":"loadbalancer-controller"},"host":"kube-node-1"},"tag":"kubernetes.var.log.containers.loadbalancer-controller-wn9ws_kube-system_loadbalancer-claim-controller-02fa6a33b7feb47e7508bd2dfea8fe47a9edfc4ce46a8539e1f7219ad44bea51.log","@timestamp":"2016-12-08T07:02:42+00:00"}]}

org.elasticsearch.index.mapper.MapperParsingException: failed to parse [kubernetes.labels]

at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:409)

at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:554)

at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:487)

at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:554)

at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:487)

at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)

at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)

at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:453)

at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:432)

at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:149)

at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:515)

at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:422)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: unknown property [run]

at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateFieldForString(StringFieldMapper.java:331)

at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:277)

at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:399)

... 14 more




My configuration in td-agent.conf as follow:

<match fluent.**>
  type null
</match>

<source>
  type tail
  path /var/log/containers/*.log
  pos_file /var/log/es-containers.log.pos
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  tag kubernetes.*
  format json
  read_from_head true
  enable_watch_timer false
  refresh_interval 10s
  read_lines_limit 1000
</source>

<source>
  type tail
  format /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
  time_format %Y-%m-%d %H:%M:%S
  path /var/log/salt/minion
  pos_file /var/log/es-salt.pos
  tag salt
</source>

<source>
  type tail
  format syslog
  path /var/log/startupscript.log
  pos_file /var/log/es-startupscript.log.pos
  tag startupscript
</source>

<source>
  type tail
  format /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  path /var/log/docker.log
  pos_file /var/log/es-docker.log.pos
  tag docker
</source>

<source>
  type tail
  # Not parsing this, because it doesn't have anything particularly useful to
  # parse out of it (like severities).
  format none
  path /var/log/etcd.log
  pos_file /var/log/es-etcd.log.pos
  tag etcd
</source>

<source>
  type tail
  format multiline
  format_firstline /^\w\d{4}/
  format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
  time_format %m%d %H:%M:%S.%N
  path /var/log/kubelet.log
  pos_file /var/log/es-kubelet.log.pos
  tag kubelet
</source>

<source>
  type tail
  format multiline
  format_firstline /^\w\d{4}/
  format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
  time_format %m%d %H:%M:%S.%N
  path /var/log/kube-apiserver.log
  pos_file /var/log/es-kube-apiserver.log.pos
  tag kube-apiserver
</source>

<source>
  type tail
  format multiline
  format_firstline /^\w\d{4}/
  format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
  time_format %m%d %H:%M:%S.%N
  path /var/log/kube-controller-manager.log
  pos_file /var/log/es-kube-controller-manager.log.pos
  tag kube-controller-manager
</source>

<source>
  type tail
  format multiline
  format_firstline /^\w\d{4}/
  format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
  time_format %m%d %H:%M:%S.%N
  path /var/log/kube-scheduler.log
  pos_file /var/log/es-kube-scheduler.log.pos
  tag kube-scheduler
</source>

<filter kubernetes.**>
  type kubernetes_metadata
</filter>

<source>
  type tail
  format multiline
  multiline_flush_interval 5s
  format_firstline /^\w\d{4}/
  format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
  time_format %m%d %H:%M:%S.%N
  path /var/log/glbc.log
  pos_file /var/log/es-glbc.log.pos
  tag glbc
</source>

<source>
  type tail
  format multiline
  multiline_flush_interval 5s
  format_firstline /^\w\d{4}/
  format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
  time_format %m%d %H:%M:%S.%N
  path /var/log/cluster-autoscaler.log
  pos_file /var/log/es-cluster-autoscaler.log.pos
  tag cluster-autoscaler
</source>

<match **>
  type copy
  <store>
    type elasticsearch
    log_level info
    include_tag_key true
    host elasticsearch-logging
    port 9200
    logstash_format true
    # Set the chunk limit the same as for fluentd-gcp.
    buffer_chunk_limit 4M
    # Cap buffer memory usage to 4MiB/chunk * 512 chunks = 2048 MiB
    buffer_queue_limit 512
    #The interval between data flushes.
    flush_interval 1s
    #The number of threads to flush the buffer.
    num_threads 4
    # Never wait longer than 5 minutes between retries.
    max_retry_wait 30
    # Disable the limit on the number of retries (retry forever).
    disable_retry_limit
    read_timeout 300s
    open_timeout 300s
  </store>

</match>



While I use  the same configuration on td-agent version 0.12.19 (the Image is gcr.io/google_containers/google_containers_fluentd-elasticsearch:1.17 ) , no error about that. So what is wrong with it?




My fluentd image build as follow:

(1) Dockerfile 

MAINTAINER Alex Robinson "ar...@google.com"
MAINTAINER Jimmi Dyson "jimmi...@gmail.com"

# Ensure there are enough file descriptors for running Fluentd.
RUN ulimit -n 65536

# Disable prompts from apt.
ENV DEBIAN_FRONTEND noninteractive

# Copy the Fluentd configuration file.
COPY td-agent.conf /etc/td-agent/td-agent.conf

COPY build.sh /tmp/build.sh
RUN /tmp/build.sh

# Run the Fluentd service.
ENTRYPOINT ["td-agent"]


(2) build.sh

#!/bin/sh
# Install prerequisites.
apt-get update

apt-get install -y -q --no-install-recommends \
  curl ca-certificates make g++ sudo bash net-tools

# Install Fluentd.

# Change the default user and group to root.
# Needed to allow access to /var/log/docker/... files.
sed -i -e "s/USER=td-agent/USER=root/" -e "s/GROUP=td-agent/GROUP=root/" /etc/init.d/td-agent

# Install the Elasticsearch Fluentd plug-in.
td-agent-gem install --no-document fluent-plugin-kubernetes_metadata_filter -v 0.24.0
td-agent-gem install --no-document fluent-plugin-elasticsearch -v 1.5.0
td-agent-gem install --no-document fluent-plugin-webhdfs
td-agent-gem install --no-document fluent-plugin-copy_ex

# Remove docs and postgres references
rm -rf /opt/td-agent/embedded/share/doc \
  /opt/td-agent/embedded/share/gtk-doc \
  /opt/td-agent/embedded/lib/postgresql \
  /opt/td-agent/embedded/bin/postgres \
  /opt/td-agent/embedded/share/postgresql

apt-get remove -y make g++
apt-get autoremove -y
apt-get clean -y

rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

Rich Megginson

unread,
Dec 8, 2016, 9:12:51 AM12/8/16
to flu...@googlegroups.com
On 12/08/2016 01:37 AM, zhuzhen...@gmail.com wrote:
> Hi, groups,
> I am using EFK logging stack on kubernetes cluster. If I don't
> comment out the following 3 lines in td-agent.conf:
> <filter kubernetes.**>
> type kubernetes_metadata
> </filter>

You need to use "de_dot true":

* `de_dot` - replace dots in labels with configured `de_dot_separator`,
required for ElasticSearch 2.x compatibility (default: `true`)
* `de_dot_separator` - separator to use if `de_dot` is enabled (default:
`_`)
> --
> You received this message because you are subscribed to the Google
> Groups "Fluentd Google Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to fluentd+u...@googlegroups.com
> <mailto:fluentd+u...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.


zhuzhen...@gmail.com

unread,
Dec 9, 2016, 1:17:30 AM12/9/16
to Fluentd Google Group
Hi,  Rich
     Thanks for your reply.
     Starting from your reply, I read about de_dot at https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter
     Then I added "de_dot" to flunetd config file td-agent.conf, see follow, but it didn't work. Any detail config or information about this?

<filter kubernetes.**>
  type kubernetes_metadata
  de_dot true
</filter>


  Thanks.
  Regards.

  Zhuzhen Wang



在 2016年12月8日星期四 UTC+8下午10:12:51，Rich Megginson写道：

Rich Megginson

unread,
Dec 9, 2016, 10:33:20 AM12/9/16
to flu...@googlegroups.com
On 12/08/2016 11:17 PM, zhuzhen...@gmail.com wrote:
> Hi, Rich
> Thanks for your reply.
> Starting from your reply, I read about de_dot at
> https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter
> Then I added "de_dot" to flunetd config file td-agent.conf, see
> follow, but it didn't work. Any detail config or information about this?
>

No, unfortunately. I have seen a problem when there are dots in the
kubernetes.labels field names, but in your case there aren't any. So I
have no idea why Elasticsearch is complaining about this:

> org.elasticsearch.index.mapper.MapperParsingException: failed to
parse [kubernetes.labels]

When the value is this: "labels":{"run":"loadbalancer-controller"}

It is a MapperParser exception - do you have an index template? What
version of ES are you using?

> <filter kubernetes.**>
> type kubernetes_metadata
> de_dot true
> </filter>
>
>
> Thanks.
> Regards.
>
> Zhuzhen Wang
>
>
>
> 在 2016年12月8日星期四 UTC+8下午10:12:51，Rich Megginson写道：
>
> On 12/08/2016 01:37 AM, zhuzhen...@gmail.com <javascript:> wrote:
> > Hi, groups,
> > I am using EFK logging stack on kubernetes cluster. If I don't
> > comment out the following 3 lines in td-agent.conf:
> > <filter kubernetes.**>
> > type kubernetes_metadata
> > </filter>
>
> You need to use "de_dot true":
>
> * `de_dot` - replace dots in labels with configured
> `de_dot_separator`,
> required for ElasticSearch 2.x compatibility (default: `true`)
> * `de_dot_separator` - separator to use if `de_dot` is enabled
> (default:
> `_`)
>
> >
> > I'll get ES warnning like this (my td-agent version is 0.12.29 ):
> >
> > [2016-12-08 07:16:56,802][DEBUG][action.bulk ] [Eleggua]
> > [logstash-2016.12.08][3] failed to execute bulk item (index) index
> > {[logstash-2016.12.08][fluentd][AVjdSsJc-DLL_mWZ7vgs],
> > source[{"log":"E1208 15:02:42.196089 1 reflector.go:214]
> >
> k8s.io/contrib/ingress-admin/loadbalancer-controller/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:109
> <http://k8s.io/contrib/ingress-admin/loadbalancer-controller/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:109>:
> <http://gcr.io/google_containers/google_containers_fluentd-elasticsearch:1.17>
>
> > ) , no error about that. So what is wrong with it?
> >
> >
> >
> >
> > My fluentd image build as follow:
> >
> > (1) Dockerfile
> >
> > FROM gcr.io/google_containers/ubuntu-slim:0.4
> <http://gcr.io/google_containers/ubuntu-slim:0.4>
> > MAINTAINER Alex Robinson "ar...@google.com <javascript:>"
> > MAINTAINER Jimmi Dyson "jimmi...@gmail.com <javascript:>"
> > an email to fluentd+u...@googlegroups.com <javascript:>
> > <mailto:fluentd+u...@googlegroups.com <javascript:>>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.

Rich Megginson

unread,
Dec 9, 2016, 10:35:52 AM12/9/16
to flu...@googlegroups.com
On 12/09/2016 08:33 AM, Rich Megginson wrote:
> On 12/08/2016 11:17 PM, zhuzhen...@gmail.com wrote:
>> Hi, Rich
>> Thanks for your reply.
>> Starting from your reply, I read about de_dot at
>> https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter
>> Then I added "de_dot" to flunetd config file td-agent.conf, see
>> follow, but it didn't work. Any detail config or information about this?
>>
>
> No, unfortunately. I have seen a problem when there are dots in the
> kubernetes.labels field names, but in your case there aren't any. So I
> have no idea why Elasticsearch is complaining about this:
>
> > org.elasticsearch.index.mapper.MapperParsingException: failed to
> parse [kubernetes.labels]
>
> When the value is this: "labels":{"run":"loadbalancer-controller"}
>
> It is a MapperParser exception - do you have an index template? What
> version of ES are you using?
>

Also - try using curl (or some other web client) to submit records to
ES, outside of fluentd. If you can get it working outside of fluentd,
you can then try to get it working within fluentd.

zhuzhen...@gmail.com

unread,
Dec 11, 2016, 9:22:58 PM12/11/16
to Fluentd Google Group
Hi, Rich
   Thanks for your reply first.
  
   My ES version is  1.5.2.


   My index templates as follow:

{
  "template_k8s_logstash" : {
    "template" : "logstash-*",
    "settings" : {
      "index.refresh_interval" : "5s"
    },
    "mappings" : {
      "_default_" : {
        "dynamic_templates" : [ {
          "kubernetes_field" : {
            "path_match" : "kubernetes.*",
            "mapping" : {
              "type" : "string",
              "index" : "not_analyzed"
            }
          }
        } ]
      }
    }
  }
}



在 2016年12月9日星期五 UTC+8下午11:33:20，Rich Megginson写道：

Rich Megginson

unread,
Dec 11, 2016, 10:45:10 PM12/11/16
to flu...@googlegroups.com
On 12/11/2016 07:22 PM, zhuzhen...@gmail.com wrote:
> Hi, Rich
> Thanks for your reply first.
> My ES version is 1.5.2.
>
>
> My index templates as follow:
>
> {
> "template_k8s_logstash" : {
> "template" : "logstash-*",
> "settings" : {
> "index.refresh_interval" : "5s"
> },
> "mappings" : {
> "_default_" : {
> "dynamic_templates" : [ {
> "kubernetes_field" : {
> "path_match" : "kubernetes.*",
> "mapping" : {
> "type" : "string",
> "index" : "not_analyzed"
> }
> }
> } ]
> }
> }
> }
> }

Not sure. You should probably ask on an Elasticsearch forum.

>
>
>
> 在 2016年12月9日星期五 UTC+8下午11:33:20，Rich Megginson写道：
>
> <http://k8s.io/contrib/ingress-admin/loadbalancer-controller/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:109
> <http://gcr.io/google_containers/google_containers_fluentd-elasticsearch:1.17
> <http://gcr.io/google_containers/google_containers_fluentd-elasticsearch:1.17>>
>
> >
> > > ) , no error about that. So what is wrong with it?
> > >
> > >
> > >
> > >
> > > My fluentd image build as follow:
> > >
> > > (1) Dockerfile
> > >
> > > FROM gcr.io/google_containers/ubuntu-slim:0.4
> <http://gcr.io/google_containers/ubuntu-slim:0.4>
> > <http://gcr.io/google_containers/ubuntu-slim:0.4
Reply all
Reply to author
Forward
0 new messages