Fluentd with MySQL Out Bulk Parsing and SSL Use
103 views
Skip to first unread message
Sahil Gupta
Apr 8, 2016, 3:46:53 PM4/8/16
to Fluentd Google Group
Hello,
I am new to Fluentd environment looking for general guidance on using Fluentd with MySQL. So, here is what I am trying to do:
Fluentd is receiving log events from source. It contains events in the format mentioned below. I am trying to store only the value for key "message" in a MySQL instance on a separate host. I saw that out_mysql_bulk may be used to achieve this: https://github.com/tagomoris/fluent-plugin-mysql/blob/master/lib/fluent/plugin/out_mysql_bulk.rb
2016-04-01 00:00:00 -0400 ident.abc : {"time":"Apr 1 00:00:00","host":"test","ident":"abc","log_time":"1 Apr 2016 00:10:00 GMT","message":"0,20.0,430"}
The "message" key above has the value: "0,20.0,430". This value in turn are values for static headers/columns e.g. key1,key2,key3 where key1 = 0, key2 = 20.0, key3 = 430.
1. But unfortunately this plugin cannot be used to communicate over SSL. How can Fluentd communicate with MySQL to perform such insert operations while enabling SSL? If there is no single plugin which can do this, is there a combination of plugins or setup within Fluentd which can achieve this?
2. How can Fluentd be used to parse key "message" and break it's value into Key-Value pairs i.e. convert "message" with value: "0,20.0,430" to "key1":"0", "key2":"20.0","key3":"430"? This way out_mysql_bulk plugin can be used to insert into table with columns key1, key2, key3.
Any advice will be appreciated. Thank you!
I am new to Fluentd environment looking for general guidance on using Fluentd with MySQL. So, here is what I am trying to do:
Fluentd is receiving log events from source. It contains events in the format mentioned below. I am trying to store only the value for key "message" in a MySQL instance on a separate host. I saw that out_mysql_bulk may be used to achieve this: https://github.com/tagomoris/fluent-plugin-mysql/blob/master/lib/fluent/plugin/out_mysql_bulk.rb
2016-04-01 00:00:00 -0400 ident.abc : {"time":"Apr 1 00:00:00","host":"test","ident":"abc","log_time":"1 Apr 2016 00:10:00 GMT","message":"0,20.0,430"}
The "message" key above has the value: "0,20.0,430". This value in turn are values for static headers/columns e.g. key1,key2,key3 where key1 = 0, key2 = 20.0, key3 = 430.
1. But unfortunately this plugin cannot be used to communicate over SSL. How can Fluentd communicate with MySQL to perform such insert operations while enabling SSL? If there is no single plugin which can do this, is there a combination of plugins or setup within Fluentd which can achieve this?
2. How can Fluentd be used to parse key "message" and break it's value into Key-Value pairs i.e. convert "message" with value: "0,20.0,430" to "key1":"0", "key2":"20.0","key3":"430"? This way out_mysql_bulk plugin can be used to insert into table with columns key1, key2, key3.
Any advice will be appreciated. Thank you!
Mr. Fiber
Apr 8, 2016, 7:20:31 PM4/8/16
to Fluentd Google Group
> 1. But unfortunately this plugin cannot be used to communicate over SSL. How can Fluentd communicate with MySQL to perform such insert operations while enabling SSL?
Hmm... it seems first SSL access request for mysql plugin.
Please open an issue on fluent-plugin-mysql.
AFAIK, mysql2 gem used by fluent-plugin-mysql supports SSL so
fluent-plugin-mysql can support SSL.
> 2. How can Fluentd be used to parse key "message" and break it's value into Key-Value pairs i.e. convert "message" with value: "0,20.0,430" to "key1":"0", "key2":"20.0","key3":"430"?
Using parser filter with csv format may work.
Masahiro
--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages