Can I have record key starts with @?

[임정택]

Hello.

First of all, I'm using Fluentd with creating statistics, log search by ES, etc.

Thanks for all your great works!

I have a question.

My favorite plugin 'fluent-plugin-elasticsearch' has an action related to @timestamp key.

But I'm having a hard time to define key to @timestamp.

1. record_reformer

Below is failed from latest release of Fluentd.

    <record>

      @timestamp ${time_year}-${time_month}-${time_day}T${time_hour}:${time_minute}:${time_second}.${time_millisecond}+09:00

    </record>

Fluentd v10 Config Parser doesn't allow @ from line start. So it says "parse error".

2. parser

format /^(?<@timestamp>[0-9]{4}-[0-9]{2}[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{3})\s+~blabla~/

It fails with complaining @ is not acceptable for group name.

How can I use @timestamp key? Is there another plugin that renames key with @timestamp with respecting Fluentd v10 Config Parser?

Thanks in advance.

Regards.

Jungtaek Lim (HeartSaVioR)

[Masahiro Nakagawa]

> First of all, I'm using Fluentd with creating statistics, log search by ES, etc.

Thanks for all your great works!

Thanks!

> Fluentd v10 Config Parser doesn't allow @ from line start. So it says "parse error".

I first knew this behaviour...

How about using --use-v1-config? Do you have a concern?

> It fails with complaining @ is not acceptable for group name.

It seems the limitation of Oniguruma regular expression engine.

I am asking Ruby engineers how to avoid this limitation.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[임정택]

Thanks for feedback!

Actually 2 is OK. I already thought it could be limitation of Ruby.

I'm really stuck on 1.

I'm a newbie of v1 config, so I wish to see your explanation.

What's difference of current and v1?

2014년 6월 12일 목요일 오후 3시 45분 28초 UTC+9, repeatedly 님의 말:

[Masahiro Nakagawa]

> What's difference of current and v1?

Here!

http://docs.fluentd.org/articles/config-file#v1-format

Almost structures are same as before.

The considering point is handling string type with escape character.

[임정택]

I'm glad to hear that option and your kind link!

But, unfortunately my latest td-agent on ubuntu doesn't have that option.

(It's based on fluentd-0.10.45)

Is there something wrong? Or it's not released yet?

Should I use source of fluentd abandoning advantage of package?

2014년 6월 12일 목요일 오후 4시 46분 43초 UTC+9, repeatedly 님의 말:

[Masahiro Nakagawa]

Ah, I see...

There are several approaches.

1. update fluentd version by /path/to/td-agent/fluent-gem update fluentd

2. use td-agent 2.0 package but this is still experimental. Some users already try this version.

    https://groups.google.com/forum/#!topic/fluentd/JXR2hmHDCZU

3. wait td-agent package update but it needs more times. Sorry...

4. use fluentd directly

[임정택]

Oh... OK. I'll try it.

Thanks again for your kind response.

Regards.

Jungtaek Lim (HeartSaVioR)

2014년 6월 12일 목요일 오후 5시 21분 15초 UTC+9, repeatedly 님의 말:

[임정택]

Finally I found other solution - use fluent-plugin-rename-key.

It's one chain longer, but it could be easy solution for current situation.

(I cannot apply all approaches because it's for commercial so distribution and operation should be easy.)

2014년 6월 12일 목요일 오후 5시 21분 15초 UTC+9, repeatedly 님의 말:

Ah, I see...