How to config the ES index and mapping by Fluentd ?

[Chen Wetry]

Hi, I'm Quinn, I use Fluentd config file to create an index and setting the mapping, and I want the index can be auto create datetime for everyday, like test-index-2018.05.17....test-index-2018.05.18. Blow is my fluentd detail config, I can't be works, so anyone can help me find what problem? 

<source>

  @type tail

  path /data/test/logs/test-index.log

  tag test.index

  pos_file /var/log/td-agent/test.index.pos 

  <parse>

    @type ltsv

  </parse>

</source>

<match test.index>

  @type elasticsearch

  host  xxx

  port xxx

  user xxx

  password xxx

  include_tag_key true

  tag_key @log_name

  logstash_format true

  logstash_prefix test-index

  template_overwrite true 

  templates { "test-index-*": "/etc/td-agent/mapping.json" }

  flush_interval 10s

  buffer_type file

  buffer_chunk_limit 8M

  buffer_path /var/log/td-agent/buffer/test-index

</match>

-----------------------------------------------------------------------

mapping.json 

{

    "mappings" : {

      "fluentd" : {

        "properties" : {

          "host" : {

            "type" : "keyword"

          },

          "req" : {

            "type" : "long"

          },

          "status" : {

            "type" : "integer"

          },

          "timestamp" : {

            "type" : "date",

            "format" : "epoch_second"

          }

        }

      }

  },

  "template": "test-index-*"

}

[Mr. Fiber]

I'm not sure why above config doesn't work.

How about using https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html ?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chen Wetry]

Yeah, I have tried use create the template first, in the template specify the mapping structure, and then create the index, it can be work, but can't create the new index in the next day, for example create test-idnex-2018.05.25 is ok, but can't auto create the test-index-2018.05.26 in tomorrow. 

在 2018年5月24日星期四 UTC+8下午4:59:51，repeatedly写道：

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Thomas Goeller]

Try this config:

<match test.index>

    @type elasticsearch

    @log_level warn # to see what’s going on with your inserts

    logstash_format true

    logstash_prefix test

    logstash_dateformat %Y-%m-%d

    host your_es_host

    .

    .

    .

</match>

This will created an index with name scheme like ’test-2018-05-25' when day/date changes.

Then using curator (https://www.elastic.co/guide/en/elasticsearch/client/curator/5.4/index.html)

you can easily manage your outdated old indices - just as a tip.

--

Mit freundlichen Grüßen                                                           

Tom Göller

Python Developer & DevOp

 

ABOUT   YOU° GmbH

Domstraße 10

20095 Hamburg

 

Telefon: +49. 40. 638 569 100 

Telefax: +49. 40. 638 569 499

 

Email: Thomas.Goeller@project-collins.com

www.aboutyou.de

...............................................................................................................

ABOUT YOU GmbH                                                                                   

Amtsgericht Hamburg HR B 120869                                                                                                

Geschäftsführer: Sebastian Betz, Tarek Müller, Hannes Wiese

 

Please consider the environment before printing this email.