send collection of logs to kinesis stream

[David Gutierrez Arias]

I am trying to send a collection of logs with a window of 5 mins to kinesis stream in order to analyse them. I would like to know if it is possible?

this is my conf.

<source>
        @type tail
        path /path/to/log
        pos_file /var/log/td-agent/td-agent-test-1.log
        format multiline
        format_firstline /[0-9]+-[0-9]+-[0-9]+/
        format1 /^(?<timestamp>[0-9]+-[0-9]+-[0-9]+\s+[0-9]+:[0-9]+:[0-9]+,[0-9]+)\s+(?<log_level>[a-zA-Z]+)\s+\[(?<class>.*)\]\s+\((?<thread_pool>.*)\)\s+(?<log_message>.*)/
        keep_time_key true
        time_key timestamp
        time_format %Y-%m-%d %H:%M:%S,%L # %L
        tag kinesis-test.log
        multiline_flush_interval 2s
</source>

## Send Data to Kinesis Stream
<match *.**>
        @type copy
        <store>
               @type kinesis_producer
               region us-east-1
               stream_name test
               include_time_key true
               buffer_chunk_limit 128mb
               buffer_queue_limit 1024
               flush_interval 300
               num_threads 32
               disable_retry_limit true
        </store>
</match>

thanks,
David

[Mr. Fiber]

What is the problem with your configuration?

Failed to send logs to kinesis?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Gutierrez Arias]

I didn't get to decode record of 5 min of logs in the kinesis. After reading the docs,I realized that 1 record in the kinesis only support 1mb . But I want to analyse 5 mins of logs. Do you know how can I analyse 5 mins of logs using kinesis?

[Mr. Fiber]

> But I want to analyse 5 mins of logs. Do you know how can I analyse 5 mins of logs using kinesis?

I don't know.

kinesis stream is a managed queue so it seems the outside of kinesis stream.

Analysis itself is done by kinesis application or other system.

If you need to analyse data in the queue, maybe, you need to re-design your architecture.

On Wed, Sep 6, 2017 at 1:41 AM, David Gutierrez Arias <david...@gmail.com> wrote:

I didn't get to decode record of 5 min of logs in the kinesis. After reading the docs,I realized that 1 record in the kinesis only support 1mb . But I want to analyse 5 mins of logs. Do you know how can I analyse 5 mins of logs using kinesis?

--