Can i write a Java plugin for fluentD

[Sandy]

Hi Folks,

I'm very new to fluentd plugin development.

Can i write a pure java plugin for Fluentd. If yes can anybody tell me how to start with it?

Java plugin meaning. I have a database for which fluentd needs to output. I want to write plugin in java which is currently not available.

Regards,

Sandy

[Masahiro Nakagawa]

Hi

> Can i write a pure java plugin for Fluentd. If yes can anybody tell me how to start with it?

Does this mean you want to write out_xxx plugin in Java?

It is now impossible.

In such case, the user uses out_exec plugin for other language.

http://docs.fluentd.org/articles/out_exec

Thanks,

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sandy]

Hi Masahiro,

Thanks for the reply. Yes my intention was to write out_xxx plugin in java.

okay I will look into out_exec plugin and see if i can solve my issue.

[Sandy]

Hi Masahiro,

I tried with out_exec. But it doesn't seems to be working. It's giving some exception.

What i tried.

Configuration file:

<source>

  type tail

  format /(?<log>.*)/

  path /tmp/sandy.txt

  pos_file /tmp/sandy_position.txt

  tag log

</source>

<match log>

  type exec

  command cat arg

  keys k1,k2,k3

  tag_key k1

  time_key k2

  time_format %Y-%m-%d %H:%M:%S

  buffer_type file

  buffer_path /tmp/sandy_fluentd.buffer

  flush_interval 5s

</match>

Format for source::  format /(?<log>.*)/ i obtained by using Fluentular.

When an entry in the source file  ==> tmp/sandy.txt is made. Nothing seems to be output.

Can some one guide me where am i going wrong? I don't know where to see the output also.

I assumed it would be displayed on the command line like anyother cat command when an entry in source file was made( My assumption) but nothing displayed.

Is my assumption wrong? Am i looking at wrong place.

Please some one try to guide me in wrong direction.

[Sandy]

***typo*** Mistake

Please some one try to guide me in correct direction.

[Masahiro Nakagawa]

I want to figure out where the problem lies.

Could you put following match instead of your match?

<match log>

  type copy

  <store>

    type stdout

  </store>

  <store>

    type exec

    command cat arg

    keys k1,k2,k3

    tag_key k1

    time_key k2

    time_format %Y-%m-%d %H:%M:%S

    buffer_type file

    buffer_path /tmp/sandy_fluentd.buffer

    flush_interval 5s

  </store>

</match>

If 'in_tail' has no problem, 'out_stdout' puts tailed logs to stdout and 'out_exec' has a problem.

Otherwise 'in_tail' has a problem. You should check format option.

Note that in_tail doesn't read the existence content at first time.

You should append new line to your watching file.

[Sandy]

Hi Masahiro,

With the match tag you gave, I was successfully able to view the appended contents of my file under /var/log/td-agent/td-agent.log.

But however, exec i guess didn't returned any success. It was giving me some error in the same log file:

Please find the below exceptions:

 

  2014-03-12 14:43:34 +0530 [warn]: temporarily failed to flush the buffer. next_retry=2014-03-12 14:43:34 +0530 error_class="RuntimeError" error="command returns 256: cat arg /tmp/sandy_fluentd.buffer.20140312.q4f4653fe989d36f7.log" instance=70082048472980

  2014-03-12 14:43:34 +0530 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.41/lib/fluent/plugin/out_exec.rb:82:in `write'

  2014-03-12 14:43:34 +0530 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.41/lib/fluent/buffer.rb:292:in `write_chunk'

  2014-03-12 14:43:34 +0530 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.41/lib/fluent/buffer.rb:272:in `pop'

  2014-03-12 14:43:34 +0530 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.41/lib/fluent/output.rb:305:in `try_flush'

  2014-03-12 14:43:34 +0530 [warn]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.41/lib/fluent/output.rb:131:in `run'

Then i removed arg from the match tag "command cat arg" realizing that by default exec gives as the last parameter the file location.

After that above error stopped. But still i wonder where the output of the exec is?

Where should i see the output of exec ? Command line or specific file like eg:(/var/log/td-agent/td-agent.log)?

[Sandy]

Where does fluentD puts all the error loggging or if plugin is mis configured not working reports to? So that i can check in detail about that and see if i can get any help to succeed with out_exec plugin.

[Masahiro Nakagawa]

> Then i removed arg from the match tag "command cat arg" realizing that by default exec gives as the last parameter the file location. After that above error stopped.

`cat` command returns the error when argument file is not exist.

So this behaviour is correct.

> Where should i see the output of exec ? Command line or specific file like 

You can get the output by adding `> foo.log 2>&1` to your command parameter.

out_exec can't detect configured command outputs the logs or not.

This is why out_exec doesn't support stdout / stderr capturing.

If configured command never output the logs, out_exec will block the program.

[Masahiro Nakagawa]

Fluentd and plugins use stdout by default.

If the user passes log file option, logs are output to its log file.

In td-agent, /var/log/td-agent/td-agent.log

[Sandy]

Hi,

Specifying the output file it worked. But one question, i'm not able to view the contents of the file which i'm trying to see.

But it can be viewed from the log location that you had mentioned using another store.

How do i do that so that my new contents along with tag comes in the file which i mention for cat command.?

Thanks.

[Masahiro Nakagawa]

Is your requirement "I want to see the file content before executes the command for debug"?

[Sandy]

My assumptions about the exec plugin:

Input: File content getting updated.

Output: As soon as the file under the observation gets modified, fluentd plugin starts emitting the contents in the form of the matched_tag to values. meaning the command would be executed as part of match tag.

What was my exepctation was, as part of the command i just do cat of the file which exec plugin passes as last argument. which i assumed would have my updated file contents along with the key type i define:

Eg:

Input file updates with content

fluentd fluent fluent

I have shell script which i call in exec just outputs the contents of the last passed argumented file name from exec.

I view the log file where the command output would be written and i see only something like this:

log 2014-03-13 06:20:52 log 2014-03-13 06:20:52 log 2014-03-13 06:20:52 log 2014-03-13 06:20:52

Where as in the /var/log/td-agent/td-agent.log i can see.

2014-03-13 11:50:52 +0530 log: {"log":"fluentd fluent5 fluent1"}

2014-03-13 11:50:52 +0530 log: {"log":"fluentd fluent6 fluent2"}

2014-03-13 11:50:52 +0530 log: {"log":"fluentd fluent7 fluent3"}

2014-03-13 11:50:52 +0530 log: {"log":"fluentd fluent8 fluent4"}

Isn't that expected similar thing from exec plugin that intermediate file path which is passed to command should be same as the log output?

[Kiyoto Tamura]

Hi Sandy,

Can you paste either here on GitHub Gist the following?

1. Your Fluentd configuration file

2. Your shell script (I assume this is different from cat)

3. What you see in stdout/stderr

4. What you see in /var/log/td-agent/td-agent.log

5. What you EXPECT to see but are NOT seeing.

Thanks!

Kiyoto

Check out Fluentd, the open source data collector for high-volume data streams

[Sandeep]

Hi Kiyoto,

Please find the required files attached in GitHub Gist under the path: https://gist.github.com/anonymous/9523266

In the td-agent.log we can see entries like below:

2014-03-13 12:41:28 +0530 log: {"log":"fluentd fluent5 fluent1"}

2014-03-13 12:41:28 +0530 log: {"log":"fluentd fluent6 fluent2"}

2014-03-13 12:41:28 +0530 log: {"log":"fluentd fluent7 fluent3"}

2014-03-13 12:41:28 +0530 log: {"log":"fluentd fluent8 fluent4"}

What was i expecting was that exec plugin would output similar to the td-agent.log informations in the file that i'm catting in the shell script and got the below output:

Hello World

Number of Arguments ==> 1

Arguments List ==>  /tmp/sandy_fluentd.buffer.20140313.q4f477a9741c0660f.log

log 2014-03-13 07:11:28 log 2014-03-13 07:11:28 log 2014-03-13 07:11:28 log 2014-03-13 07:11:28

Please let me know if my understand is not correct.

[Masahiro Nakagawa]

> In the td-agent.log we can see entries like below:

You want similar content in passed the file,

you should specify correct keys in exec configuration like below:

    keys time,tag,log

    tag_key tag

    time_key time

out_exec automatically adds "\n" to each event.

So concat logs without "\n" is weird... 'log 2014-03-13 07:11:28 log 2014-03-13 07:11:28 log'

My result is below:

  <source>

    type tail

    format /(?<log>.*)/

    path /tmp/sandy.txt

    pos_file /tmp/sandy_position.txt

    tag log

  </source>

  <match log>

    type exec

    command cat > /Users/repeatedly/tmp/fluentd/sandy_log 2>&1

    keys time,tag,log

    tag_key tag

    time_key time

    time_format %Y-%m-%d %H:%M:%S

    buffer_type file

    buffer_path /tmp/sandy_fluentd.buffer

    flush_interval 5s

  </match>

command:

% echo 'fluentd fluent fluent' >> /tmp/sandy.txt

% echo 'fluentd fluent fluent' >> /tmp/sandy.txt

% cat sandy_log

2014-03-13 08:14:29     log     fluentd fluent fluent

2014-03-13 08:14:29     log     fluentd fluent fluent

[Sandeep]

Hi,

Thanks a lot for all the help. I'm now able to view the contents as well with the exec plugin.

yes exec plugin is not appending "\n" after every event. Any configuration which i need to do so that i can get each event in new line?

PS: In the test environment of yours, exec is not putting events in new line?

[Masahiro Nakagawa]

My result is default configuration and out_exec added "\n" to each event. Code is below:

https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/out_exec.rb#L63

How to print the file content in your script?

[Sandeep]

In your case with default exec plugin, it is outputting new line.i'm also using currently the default exec plugin itself. 

But it's not giving me with new line for each event.

>>How to print the file content in your script? which script you are referring to?

PS:  I have currently not developed my own plugin.

[Masahiro Nakagawa]

Ah, I didn't notice your gist includes command script and

I found your script has a problem.

> But it's not giving me with new line for each event.

It is shell problem.

By default, shell's expansion deletes all newlines.

Try following 2 commands:

% echo `cal`

% echo "`cal`" 

 

[Sandeep]

Hi,

yes after the second one, including in the shell script, I was able to get the events/values in the new lines. Thanks a lot for all the help.