Packaging and Tests

[Andrew Farrell]

Hi all,

I am new fluentd and new to ruby!

So a little bit of learning to be done.

But - I have written a plugin which works.  It is a filter plugin and I have it working in a pipeline between tail for input and webhdfs for output, viz.

 <source>

    @type tail

    path "/home/andrew/testdata.json"

    tag "test.default"

    <parse>

      @type json

    </parse>

</source>

<filter>

    @type sparkl

    keep_rawdata true

</filter>

<match test.*>

    @type webhdfs

    host "localhost"

    port 50070

    path "/user/andrew/proctestdata.json"

</match>

The plugin does the processing I want.

Here is the simple ruby source.

https://github.com/sparkl/fluent-plugin-sparkl

Two things:

1) I want to package it properly as a gem.

-- Could you point me to any instructions that specifically say how I should do this for fluentd?

I am using currently: bundle gem foo, then gem build, gem install.

This does not appear to be an appropriate workflow for use with fluentd.  Could you correct me on that, pls.

As a short term measure, I am just dropping the source (filter_sparkl.rb) into fluentd-0.14.11/lib/fluent/plugin/

2) I want to write some tests that both sanity check the code, but also demonstrate its purpose.

-- All I have, from the fluentd website, is: "read the sourcecode"!  Could you point me to any meaningful information for writing tests?

I am using Ruby 2.4.0 with FluentD 0.14

Many thanks, and much appreciated.

I hope to release many plugins to the community over time.

Cheers

Andrew Farrell

[Mr. Fiber]

Hi Andrew, 

Sorry for delay response and missing plugin development article.

1)

fluentd gem is normal rubygems, so rubygems's article is also helpful:

http://guides.rubygems.org/make-your-own-gem/

For v0.14, you can use fluent-plugin-generate command to get skelton code for fluentd plugin.

You can check your current code by this command.

http://www.fluentd.org/blog/fluentd-v0.14.13-has-been-released

Above article also shows how to publish gem.

2)

Fluentd's built-in tests are helpful.

test_filter_grep is one good example.

https://github.com/fluent/fluentd/blob/master/test/plugin/test_filter_grep.rb

If you have further quesiton, please let me know.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Farrell]

Thanks Masahiro,

I will check the links out and try again!

Many thanks

Andrew

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/pPhyWHPbELw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Dr Andrew Farrell Lead Software and Research Engineer SPARKL - Christchurch, New Zealand t 0064 22 438 4944

[Andrew Farrell]

Hi Masahiro,

Please see experimental plugin here:  

https://github.com/sparkl/fluent-plugin-sparkl

I'll submit it to rubygems.

Any advice on getting it certified by fluentd would be helpful.  I am thinking that I will need to split it into a filter plugin and an output plugin at some point for performance reasons, but I need a better understanding of fluentd under-the-hood in order to determine the best route for this.

The plugin inserts meta-data into some of the events in an event stream - its fairly deterministic in regards of performance for this.  That functionality would remain in the filter plugin.

It also periodically pushes data to a public blockchain.  This is probably better suited as an output plugin.

Any feedback is much appreciated,

Thanks

Andrew

On 4 April 2017 at 19:25, Mr. Fiber <repea...@gmail.com> wrote:

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/pPhyWHPbELw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Mr. Fiber]

>  That functionality would remain in the filter plugin.

Yes. Filter is for mutating events.

> This is probably better suited as an output plugin.

It depends on plugin. If you can separate features into filter / output, providing output plugin is good.

If not, using background thread for emitting new event in filter is one approach.