Fluentd

[anaka kiran]

  Hii

  I am new to the Fluentd, and am trying to get the logs from different server. I am doing this in node js.I want to know about how i can collect the logs from different servers .I have to specify the file path also.Like from this particular server path i have to take the logs.So how can i do that?
Can anyone please help me on that?

[Kentaro Hayashi]

Hi,

It may be useful to read "Guides and Recipes" to understand what is Fluentd and what Fluentd does

https://www.fluentd.org/guides

I hope it will help.

Regards,

2020年12月11日金曜日 18:56:34 UTC+9 anaka...@gmail.com:

[anaka kiran]

Hi,

By using Fluentd am getting the logs and storing into the redis.But here the issue is like when the new logs comes the old ones are getting deleted.So how can i append the new log data in redis?
And another one is like i have to give the data in JSONARRAY.How can i give that one?
Please help one this issue

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/vcPUUdHwYuQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluentd/16f9322d-6614-4322-9741-eb090ebb687en%40googlegroups.com.

[Kentaro Hayashi]

Hi,

> So how can i append the new log data in redis?

I'm not sure what you want, but you can specify unique insert_key_prefix explicitly. [1]

For example,  insert_key_prefix '%Y%m%d.%H%M%S' or something else to avoid conflict.

[1]

https://github.com/fluent-plugins-nursery/fluent-plugin-redis#configuration

I hope it will help

Regards,

2021年1月14日木曜日 15:00:45 UTC+9 anaka...@gmail.com:

[anaka kiran]

<source>
    @type tail
    path xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    pos_file xxxxxxxxxxxxxxxxxxxxxxxxx
    read_from_head true
    tag local
    <parse>
      @type json
    </parse>
  </source>
<match **>
    @type redis_store_seldon
    host xxxxxx
    port xxxx
    password xxxxxx
    db xx
    key_prefix order:
    key_prefix_sep :
    key_path xxxxx
    key_prefix_path xxxxx
    key_suffix :xxxx
    store_type string
    format_type json
</match>

this is my config.
what i am doing like i am reading the logs from file..once i have read from the log file then again that same log file is updating.so the new log data i have to append to the redis.
now what is happening like the new data is over writing the old data from the redis.
i have to store the old data and newly coming data to the redis.

To view this discussion on the web visit https://groups.google.com/d/msgid/fluentd/bf314650-0266-4113-af77-f803c32267c3n%40googlegroups.com.

[anaka kiran]

Did anyone know how to give?
Please give me a reply

[Kentaro Hayashi]

Hi,

I've assumed that @type redisoutput plugin [1] so I've thought that

insert_key_prefix may be useful, but @type redis_store_seldon [2]

doesn't have a similar feature.

Anyway, it seems that "key_path xxxxx" is specified, so if the value of key xxxx

is unique, it will work as expected. it is not unique, you need to modify record

such a event by record_transformer [3] something like this:

<filter **>

    @type record_transformer

   <record>

      unique_key "..." <- assign unique value

   </record>

</filter>

<match **>

  ...

  key_path unique_key

</match>

[1] https://github.com/fluent-plugins-nursery/fluent-plugin-redis

[2] https://github.com/SeldonIO/fluent-plugin-redis-store

[3] https://docs.fluentd.org/filter/record_transformer

2021年1月15日金曜日 14:24:58 UTC+9 anaka...@gmail.com:

[anaka kiran]

how can i parse a string and json?
10.10.10.165;starttime:123443545465657;post;/api/log; 200;{"uid":"xxxx","pwd":"12345"};;endtime:2334556576;{"status":"FAILURE","message":"Invalid user details."}

this is my log.
How can I parse these logs?

To view this discussion on the web visit https://groups.google.com/d/msgid/fluentd/a0231b9e-3c3b-467e-9756-005b13f16c79n%40googlegroups.com.

[anaka kiran]

And I want to take the value of uid ?
can anyone please give me a reply 

[Kentaro Hayashi]

Hi,

You can use @type regexp for such a purpose.

You need to write your own expression to match fields by named capture.

<source>

...

  <parse>

     @type regexp

     expression /^(?<address>....)/ <= MODIFY HERE

  </parse>

<source>

Then you can get {"address": ... } record.

Regards,

2021年2月9日火曜日 19:00:14 UTC+9 anaka...@gmail.com:

[anaka kiran]

Helo,
thanks for your reply.

i have tried with regexp ,but  am getting as pattern not matched

You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluentd/022448ce-0443-47d0-9fa7-527c8a1b78e1n%40googlegroups.com.

[anaka kiran]

<source>
    @type tail
    path  ......................................
    pos_file  ...............................
    read_from_head true
    tag logs
   <parse>
      @type none
   </parse>
</source>
<match .**>
   @type redis_store_seldon
     key_path  uid
     store_type string  
</match>

I am getting the logs in my key.but in the key_path am not getting the value for uid.
can any one please help me on this?

[Kentaro Hayashi]

Hi,

If you want to use uid as key_path, you must extract the key from  log in the beginning.

Thus,

1. parse your log in <parse> section with regexp and extract uid from it

2. specify key_path uid

1. It may be too ad-hoc but it is enough to explain the sample

<source>
  @type  tail
  path sample.log

  read_from_head true
  tag logs
  <parse>

    @type regexp
    # 10.10.10.165;starttime:123443545465657;post;/api/log; 200;{"uid":"xxxx","pwd":"12345"};;endtime:2334556576;{"status":"FAILURE","message":"Invalid user details."}
    # Extract address and uid, pwd from above sample.log
    expression /^(?<address>\d+\.\d+\.\d+\.\d+)\;.+\{"uid":"(?<uid>.+)","pwd":"(?<pwd>.+)"};.+/
  </parse>
</source>

Then {"address":"10.10.10.165","uid":"xxxx","pwd":"12345"} record is extracted and processed in <match **> section

<match .**>
   @type redis_store_seldon
     key_path  uid
     store_type string  
</match>

So you can refer key_path uid because the record contains "uid" as a key.

Regards,

2021年2月11日木曜日 19:33:52 UTC+9 anaka...@gmail.com: