Ask a question about td-agent/fluentd reload config file in high frequency

856 views
Skip to first unread message

Jin Aazoe

unread,
Oct 9, 2016, 11:16:12 PM10/9/16
to Fluentd Google Group
Right now I'm developing a project that export data into elasticsearch, which means every users have to generate his own config file, and after that, the td-agent/fluentd have to reload that config file.
I'm using fluentd's rpc /api/config.reload to reload ,when a user curl this address,fluentd will automatically restart.
My question is what if I have thousands of users?if they all need to generate their own config file and reload, will fluentd get stuck in reload this thousands of config files and wouldn't do its real work?

Jin Aazoe

unread,
Oct 9, 2016, 11:24:31 PM10/9/16
to Fluentd Google Group
Oh,I'm using in_http plugin and fluent-elasticsearch-plugin

在 2016年10月10日星期一 UTC+8上午11:16:12,Jin Aazoe写道:

Mr. Fiber

unread,
Oct 10, 2016, 12:28:32 AM10/10/16
to Fluentd Google Group
My question is what if I have thousands of users?if they all need to generate their own config file and reload, will fluentd get stuck in reload this thousands of config files and wouldn't do its real work?

Sorry, I can't understand your situation correctly.
Do thousands of users share one fluentd instance?


Masahiro


--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jin Aazoe

unread,
Oct 10, 2016, 1:55:43 AM10/10/16
to Fluentd Google Group
yes, they share one fluentd instance

在 2016年10月10日星期一 UTC+8下午12:28:32,repeatedly写道:
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

Jin Aazoe

unread,
Oct 10, 2016, 2:00:14 AM10/10/16
to Fluentd Google Group
for example, user A is generating a conf file ,when user B curl a rpc reload request ,and the same time ,user C is trying to export data into elasticsearch, then what will happen ?


在 2016年10月10日星期一 UTC+8下午12:28:32,repeatedly写道:
My question is what if I have thousands of users?if they all need to generate their own config file and reload, will fluentd get stuck in reload this thousands of config files and wouldn't do its real work?
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

Mr. Fiber

unread,
Oct 10, 2016, 2:12:14 AM10/10/16
to Fluentd Google Group
for example, user A is generating a conf file ,when user B curl a rpc reload request ,and the same time ,user C is trying to export data into elasticsearch, then what will happen ?

So if B changes configuration file, C's configuration is replaced with B's one, right?
Or operations are serialized into one sequence?

BTW, we, fluentd developers, don't assume such senario,
so I didn't test it. You are the first person to do that in the world.
To be honest, you may hit the problem.


To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

Jin Aazoe

unread,
Oct 10, 2016, 2:20:16 AM10/10/16
to Fluentd Google Group
in my case,user A has conf file A.conf, user B has B.conf ,and user C has C.conf,and in fluentd.conf, I use include *.conf to get A,B,C.conf files,users can only change their conf file
e...it's my first time using fluentd and certainly I don't fully understand what fluentd exactly could do, hahaha...

在 2016年10月10日星期一 UTC+8下午2:12:14,repeatedly写道:

Mr. Fiber

unread,
Oct 12, 2016, 8:05:08 PM10/12/16
to Fluentd Google Group
in my case,user A has conf file A.conf, user B has B.conf ,and user C has C.conf,and in fluentd.conf, I use include *.conf to get A,B,C.conf files,users can only change their conf file

I use include *.conf to get A,B,C.conf files,users can only change their conf file

It may work but you should validate all users use different routing rule in my configuration.
And basically, input plugin and output plugin have own thread to separate execution context.
So if 1000 users set 1 input / 1 output plugins, 2000 threads are launched in one process.
This is very heavy in general.


To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.

Jin Aazoe

unread,
Oct 17, 2016, 5:33:30 AM10/17/16
to Fluentd Google Group
OMG, I didn't know that at the beginning, thousands of threads will cost a year to reload ... maybe I will change my strategy to prevent reload,.routings are all separated,every user has his own match path,but 千の threads is really a big problem for me ,headache...
thx for the advice,any way~ :)

在 2016年10月13日星期四 UTC+8上午8:05:08,repeatedly写道:

Jin Aazoe

unread,
Oct 17, 2016, 10:20:16 PM10/17/16
to Fluentd Google Group
hah,I test 2000 config files on my server and reload fluentd but it performs fine,but I just reload, not import data... needs further research


在 2016年10月13日星期四 UTC+8上午8:05:08,repeatedly写道:
in my case,user A has conf file A.conf, user B has B.conf ,and user C has C.conf,and in fluentd.conf, I use include *.conf to get A,B,C.conf files,users can only change their conf file

Pranob Samani

unread,
Sep 17, 2019, 5:26:35 PM9/17/19
to Fluentd Google Group
Hi how are you reloading config via rpc/ curl
Reply all
Reply to author
Forward
0 new messages