storing the fluentd collector logs file on s3 bucket with hash

[NIKHIL SHARMA]

Hi,

As we are using fluentd logs to put s3 bucket we need to create hash file for every s3 bucket.

Is it possible please share the details.

Thanks 

Nikhil

[Mr. Fiber]

What does 'hash file' mean?

> --
> You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[NIKHIL SHARMA]

Hi,

How can we  make the integrity of the stored logs so the stored logs in s3 can't be edited or modified by anyone.

So is it possible if we can make hash by (sha2 or md5) of the stored logs.

Thank 

Nikhil

[Mr. Fiber]

I'm not sure your requirement correctly but

s3 plugin uploads buffered chunks to S3 without additional files.

So if you want to do that, extending s3 plugin or using lambda is better.

On Sun, Jul 24, 2016 at 3:23 PM, NIKHIL SHARMA <nikhil...@gmail.com> wrote:

Hi,

How can we  make the integrity of the stored logs so the stored logs in s3 can't be edited or modified by anyone.

So is it possible if we can make hash file of the stored logs.

Thank 

Nikhil

On Thursday, July 21, 2016 at 7:21:15 PM UTC+5:30, repeatedly wrote:

What does 'hash file' mean?

On Jul 21, 2016 6:57 PM, "NIKHIL SHARMA" <nikhil...@gmail.com> wrote:
>

> Hi,if we 

>
> As we are using fluentd logs to put s3 bucket we need to create hash file for every s3 bucket.
> Is it possible please share the details.
>
> Thanks 
> Nikhil
>
> --
> You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[NIKHIL SHARMA]

ok thanks for sharing the info i am using same s3 plugin to forward the logs but how can we maintain the integrity that stored logs on s3 can't be changed or modified after storing on our bucket.Is there any way out?

Thanks 

Nikhil

[Mr. Fiber]

> Is there any way out?

I'm not sure because it seems your data management policy.

Maybe, other place, e.g. AWS forum, is better.

Masahiro

[NIKHIL SHARMA]

ok i will check let me bit more clear that is it possible if i store the logs with any hash function(sha-1 or sha-2 or md5) to maintain the integrity of the log files stored in the bucket.

Thanks

Nikhil

[Mr. Fiber]

> is it possible if i store the logs with any hash function(sha-1 or sha-2 or md5) to maintain the integrity of the log files stored in the bucket.

To extend s3 plugin may realize it.

[NIKHIL SHARMA]

please explain bit more

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/GUEIuIgvZQo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.

[Mr. Fiber]

> please explain bit more

Write your plugin based on S3 plugin.

[Mr. Fiber]

fluent-plugin-s3's main routine is write.

So overriding this method realizes your purpose.

https://github.com/fluent/fluent-plugin-s3/blob/bb3e171bf392c2fcc8960837f96fcf6158fcdbde/lib/fluent/plugin/out_s3.rb#L188