Batch mode?

[smcm...@nvidia.com]

Hi,

In addition to the usual monitoring use cases, I'm interested in using fluentd in "batch" mode. 

This very simple example is almost what I'm looking for:

  <source>

    @type exec

    command uptime

    format tsv

    keys stdout

    tag uptime_tag

    time_format %Y-%m-%d %H:%M:%S

  </source>

  <match uptime_tag>

    @type file

    format stdout

    path /home/scott/fluent/out

    flush_at_shutdown true

  </match>

The only thing missing is that I want fluentd to exit after the one sample is collected rather than continue indefinitely.  Something like:

% td-agent -c myfile.conf --batch

I'm new to fluentd, so my apologies if this is obvious.

Thanks,

Scott

[Mr. Fiber]

Hi Scott,

Hmm... fluentd doesn't have such option because

fluentd core hard to detect one sample by default.

https://github.com/repeatedly/fluent-plugin-stdin/blob/master/lib/fluent/plugin/in_stdin.rb#L63

fluent-plugin-stdin has similar feature but this is tricky approach...

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kiyoto Tamura]

Scott,

Have you looked into Fluentd's sister project Embulk? If your goal is to periodically move files, using Embulk might be a better approach.

Like Fluentd, my colleagues at Treasure Data will be happy to help with Embulk.

Kiyoto

--

Refer a friend to Treasure Data and get paid! Ask me about our referral program =)

[smcm...@nvidia.com]

Hi,

Thank you, I was not aware of Embulk.  I'll have to check it out, but at first glance it's not quite what I'm looking for.  I'd like to use fluentd as a background data collection service, as it's intended, but occasionally "force" a data collection event from the command line.  The batch run may be a superset or a subset of the background data collection.  The key is to use the same configuration file format for both use cases.  It looks like Embulk has it's own configuration file format that is different than fluentd?

I hacked on fluentd a bit and have something that's pretty close that what I'm looking for.  I added a batch bool flag to lib/fluent/engine.rb and if set to true, skips the default_loop section in run.  I also modified lib/fluent/plugin/in_exec.rb to force everything into run rather than run_periodic if batch mode is enabled.  Finally, if batch mode is enabled, then the in_exec threads immediately join after Thread.new().  It's very crude; for example, I haven't considered yet what happens to other input plugins in batch mode.  Is this general approach something that the fluentd maintainers would consider?  If so, I can clean it up and send a pull request to get feedback.

Thanks,

Scott

[Lance N.]

You can write a shell script that fetches data and creates a field set (as a json message). I used this for miscellaneous reports by shell scripts.

<source>

  type http

  port 8999

</source>

<match post.**>

  type record_reformer

  ...

</match>

From the shell:

$ curl -X POST -d 'json={"json":"message"}' http://localhost:8999/post.text

[smcm...@nvidia.com]

On Monday, August 15, 2016 at 3:43:13 PM UTC-5, Lance N. wrote:

You can write a shell script that fetches data and creates a field set (as a json message). I used this for miscellaneous reports by shell scripts.

<source>

  type http

  port 8999

</source>

<match post.**>

  type record_reformer

  ...

</match>

From the shell:

$ curl -X POST -d 'json={"json":"message"}' http://localhost:8999/post.text

I think this is essentially the same as fluent-cat?   

Thanks,

Scott

[Lance Norskog]

I suppose so. I run fluent inside of one Docker container and post these messages from another, so I had to code it with standard Unix tools.

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/oedCS_itRYw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Lance Norskog
lance....@gmail.com
Redwood City, CA