How to "Use your (de)compression algorithm" with fluent s3 plugin

[Ashraf Mohammed]

Hi ,

We have the requirement to to compress and send files to S3 with snappy compression. I followed the doc suggestion for fluent s3 plugin and created the s3_compressor_snappy.rb algo and copied it to the fluent-se.x.x.x/lib/fluent/plugin. However with fluentd startup the newly added compressor is not registered/identified.

Can someone please suggest on how we can make the compressor to take effect?

Thanks, any help is apprceiated..

[Mr. Fiber]

Please paste your code.

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ashraf Mohammed]

Thanks for your response

Below is my code snippent

module Fluent::Plugin
  class S3Output
    class SnappyCompressor < Compressor
      S3Output.register_compressor('snappy', self)

      def initialize(options = {})
        begin
          require "snappy"
        rescue LoadError
          raise Fluent::ConfigError, "Install snappy before use snappy compressor"
        end
      end

      def ext
        ".sz"
      end

      def compress(chunk, tmp)
        w = Snappy::Writer.new(tmp)
        chunk.write_to(w)
        w.close
      end
    end
  end
end

[Mr. Fiber]

I tested your code and it worked on my env.

- code location

% ls ~/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/fluent-plugin-s3-1.1.6/lib/fluent/plugin/
in_s3.rb                       s3_compressor_gzip_command.rb  s3_compressor_lzo.rb           s3_extractor_gzip_command.rb   s3_extractor_lzo.rb
out_s3.rb                      s3_compressor_lzma2.rb         s3_compressor_snappy.rb        s3_extractor_lzma2.rb

- fluentd log

2019-01-21 15:16:38 +0900 [info]: parsing config file is succeeded path="s3_v14.conf"
2019-01-21 15:16:39 +0900 [warn]: snappy not found. Use 'text' instead
2019-01-21 15:16:39 +0900 [info]: using configuration file: <ROOT>
  <source>
    @type forward
  </source>
  <match s3.**>
    @type s3
    s3_bucket "fluent-plugin-s3-rd"
    path "test/"
    store_as "snappy"
    <assume_role_credentials>
      # ...
    </assume_role_credentials>  </match>

</ROOT>
2019-01-21 15:16:39 +0900 [info]: starting fluentd-1.3.3 pid=89697 ruby="2.4.4"
... snip ...
2019-01-21 15:16:39 +0900 [info]: gem 'fluent-plugin-s3' version '1.1.6'
... snip ...
2019-01-21 15:16:39 +0900 [info]: gem 'fluentd' version '1.3.3'
2019-01-21 15:16:39 +0900 [info]: adding match pattern="s3.**" type="s3"
2019-01-21 15:16:40 +0900 [warn]: #0 snappy not found. Use 'text' instead  # code works correctly
2019-01-21 15:16:40 +0900 [info]: adding source type="forward"
2019-01-21 15:16:40 +0900 [info]: #0 starting fluentd worker pid=89725 ppid=89697 worker=0

I assume you typo filename or something.

Could you re-check it?

[Ashraf Mohammed]

Thanks for your response, I will recheck on my end.

With my setup as well I too get below warn log and on S3, I see text files:

2019-01-21 15:16:40 +0900 [warn]: #0 snappy not found. Use 'text' instead

Do you mean, despite of this warning logs pushed to S3 are in snappy compressed form?

[Mr. Fiber]

> Do you mean, despite of this warning logs pushed to S3 are in snappy compressed form?

No. I mean code and gem location are correct.

Above log is removed after installed snappy gem on my env.

[Ashraf Mohammed]

I checked on file naming and it looks fine.

I assume, I am missing something with installing the snappy gem. Below are details of my env setup and what I did to include snappy compression:

1. We have a td-agent as fluentd aggregator setup on AWS ec2

2. Fluent S3 plugin of version 1.1.7 installed

3. Created rb script for Snappy compression

4. Placed the script under S3 plugin lib folder

5. restarted the td-agent

Am I missing any step or doing something incorrect here? Please suggest

Thanks

[Mr. Fiber]

When install snappy gem?

[Ashraf Mohammed]

Looks like I missed the install gem step, can you please let me know steps to install,?

[Mr. Fiber]

"gem install snappy" seem enough.

Of course, use /usr/sbin/td-agent-gem in td-agent.

On Mon, Jan 21, 2019 at 5:11 PM Ashraf Mohammed <mohd.as...@gmail.com> wrote:

Looks like I missed the install gem step, can you please let me know steps to install,?

[Ashraf Mohammed]

Thanks, it working fine now

I appreciate your help