Dynamic configuration update (restarting td-agent frequently?)

[wsor...@hubspot.com]

I'm thinking of using td-agent to push logs from a variety of services which will need special configuration. 

These services may be launched on a given host at any time and may stop at any time.

My question is, if I am frequently rewriting the configuration file and restarting the service, will td-agent handle this gracefully? Ie, if an upload to S3 is in progress, or remembering it's place in the log it is tailing? O

[Kiyoto Tamura]

Hi,

>Ie, if an upload to S3 is in progress, or remembering it's place in the log it is tailing? O

Yes it remembers the position, as long as your in_tail plugin (the plugin to tail a log file) has pos_file option set: http://docs.fluentd.org/articles/in_tail

Fluentd is reasonably good across restarts as long as you buffer everything ot file (http://docs.fluentd.org/articles/buf_file).

That being said, I am not familiar with use cases where restarts are frequent and anticipated.

Kiyoto

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Check out Fluentd, the open source data collector for high-volume data streams

[Whitney Sorenson]

Our use case is that we deploy services dynamically using a mesos scheduler (https://github.com/HubSpot/Singularity) to slave machines inside our cluster. We would like to send those services' logs to S3 using td-agent, as opposed to writing our own log collection and transport layer. Since services are more transient in this model, reliably handling logs is even more important.

However, since a new service may be deployed or removed from a slave machine at any time, we would need to update the config file for td-agent to add new input and output for each service running. 

I assume this would mean restarting td-agent after each configuration update. I understand that tailing using a position would be generally safe, but what about s3 uploads? Does a restart wait for certain operations to complete or does it simply kill the process? I am not sure how the output plugins would handle such events.

Since this is an unusual use case for td-agent do you think we'd be better suited to researching alternatives, building our own, or are there perhaps simple changes to better accommodate this style of service deployments (which we believe to be the very near future, ala borg, etc.)?

 

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/LhMLwOttC4A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.

[Christian Hedegaard]

As long as you use a common prefix for your tags, you could use something like forest to dynamically do all the stuff you want. I’m currently deploying forest + S3 for dynamic logging and uploading to S3.

 

https://github.com/tagomoris/fluent-plugin-forest

[Whitney Sorenson]

If I'm reading this correctly, this would potentially solve the problem of sending to N output sources, but wouldn't I still require a restart for every service to add an <in_tail> section? Still unanswered what effect this would have on output plugins.

[Satoshi Tagomori]

Hi, 

If you are talking about application logs from your code, fluent-logger and localhost fluentd(td-agent) pattern fits such environment.

 * a single fluentd listens w/ in_forward on your host

 * many application codes throws log over fluent-logger for language you use into localhost's fluentd's port 24224

   with tag of deployed service name

 * fluentd which receives collected logs sends to S3 w/ fluent-plugin-forest & fluent-plugin-s3

If you want to correct logs of apache httpd or nginx or such like middleware, you can use fluent-plugin-tail-ex with

file path globs and dynamic tags (configure refresh_interval attribute).

https://github.com/yosisa/fluent-plugin-tail-ex

2014年3月4日火曜日 22時49分48秒 UTC+9 Whitney Sorenson:

[Masahiro Nakagawa]

>  Does a restart wait for certain operations to complete or does it simply kill the process?

S3 is a buffered output plugin.  Restart behaviour of buffered output depends on your buffer plugin and configuration.

No waiting:

   - file buffer with `flush_at_shutdown false` (default is false). 

     In this case, buffered content is kept on disk and re-used after td-agent restart.

Waiting the flush logs:

  - memory buffer

  - file buffer with `flush_at_shutdown true` configuration.

If you shutdown the slave, you should set `flush_at_shutdown true`.

EC2 users use same approach.

[Kazuki Ohta]

Whitney,

We're also deploying 200+ td-agent in the cloud, where instances are always up and down. In this case, I recommend the HA configuration.

> http://docs.fluentd.org/articles/high-availability

The reason is I rarely see people adding the input plugins (forward, unix, http, syslog do the most of the job), but adding the output plugin is a common operation. By having the 2+ dedicated aggregator nodes, your edge nodes can concentrate on forwarding the data to the aggregators while you only need to change the configuration of aggregator nodes for filtering and output. You can control how to handle the logs by tagging the records.

> http://docs.fluentd.org/articles/config-file#2-ldquomatchrdquo-tell-fluentd-what-to-do

I would also recommend having really short flush_interval at the forwarder nodes (like 3sec - 5sec) to forward the logs in almost real-time. Also don't forget to send the signals when you terminate your container.

> http://docs.fluentd.org/articles/out_forward#flushinterval

> http://docs.fluentd.org/articles/signals

Thanks -K

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 

::   Kazuki Ohta - [http://www.treasuredata.com/]

::   Founder and CTO, Treasure Data, Inc

::   k...@treasure-data.com | +1-650-223-5679

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 

[Whitney Sorenson]

Thank you for all the responses. They have been very helpful in understanding the options available and determining best practices for deploying to a large cluster, and the fluentd community seems very strong.

Since I have many, many engineers writing services I do not want to change all of their root logging appenders to stream to fluentd (yet) but instead I have found that a combination of tail_ex, rewrite_tag_filter, and forest may in fact make it possible to support dynamic service configurations without dynamic updates to fluentd / td-agent.

This is great. A quick question, is there no way to combine input plugins? What is the recommended path, for example, if I wanted to use tail_ex but I needed it to support multiline (https://github.com/tomohisaota/fluent-plugin-tail-multiline). A third plugin combining the two? A patch to tail_ex?

Thank you again,

-Whitney

[Masahiro Nakagawa]

> I have found that a combination of tail_ex, rewrite_tag_filter, and forest may in fact make it possible to support dynamic service configurations without dynamic updates to fluentd / td-agent.

Good :)

>  What is the recommended path, for example, if I wanted to use tail_ex but I needed it to support multiline (https://github.com/tomohisaota/fluent-plugin-tail-multiline). A third plugin combining the two? A patch to tail_ex?

Hm. I think throw the patch supports multiline feature to tail_ex is good.

Because I have a plan which tail_ex and tail_multiline are merged into core in the future.

If tail_ex supports multiline, merge plugins becomes easier :)

[Kiyoto Tamura]

>>What is the recommended path, for example, if I wanted to use tail_ex but I needed it to support multiline (https://github.com/tomohisaota/fluent-plugin-tail-multiline). A third plugin combining the two? A patch to tail_ex?

>Hm. I think throw the patch supports multiline feature to tail_ex is good.

Because I have a plan which tail_ex and tail_multiline are merged into core in the future.

If tail_ex supports multiline, merge plugins becomes easier :)

Actually, this is a good opportunity to discuss which feature of community-contributed tail-like plugins we want to merge in to in_tail. Do we want to start a separate thread or discuss it on a GitHub Issue?

Kiyoto

[Masahiro Nakagawa]

tail_ex depends on fluent-mixin-config-placeholders.

This is a blocker for merging into core.

After introduced new configuration, this problem will be resolved.

[Whitney Sorenson]

Consider this a +1 to integrating tail_ex and multiline functionality ;)

[Masahiro Nakagawa]

One idea is removing ${UUID} from tail_ex and keeping ${hostname}.

Generating UUID requires uuid gem but getting hostname can be implemented by Socket.gethostname.

For support only ${hostname}, we can remove fluent-mixin-config-placeholders dependency.

[Masahiro Nakagawa]

Issue for this problem:

https://github.com/fluent/fluentd/issues/269

[Yuji Yamano]

We merged tail_ex into tail_multiline. Does anyone want to try it? 

It works well for us. 

https://github.com/OGIS-RI-EOS/fluent-plugin-tail-multiline/tree/feature/merge-tail-ex