Streaming data into Google Bigquery using fluentd (0.14.6) and fluent-plugin-bigquery (0.2.16)

439 views

Skip to first unread message

Alan Green

unread,

Sep 14, 2016, 2:22:08 AM9/14/16

to Fluentd Google Group

Is there possibly a compatibility issue between the Google bigquery plugin fluent-plugin-bigquery and fluentd 0.14.* ?

Soon after startup I get this error

2016-09-13 17:09:22 +0300 [error]: unexpected error error_class=ArgumentError error="wrong number of arguments (given 1, expected 0)"
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/output.rb:333:in `start'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/compat/output.rb:655:in `start'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluent-plugin-bigquery-0.2.16/lib/fluent/plugin/out_bigquery.rb:233:in `start'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/compat/call_super_mixin.rb:42:in `start'

td-agent works using the same configuration file but I need fluentd as my platform is Windows.

I've tried all the configuration options I can think of, without success.

Is there anyone in this group using fluentd 0.14.6 to stream logs into Google Bigquery?

I'm adding a fuller trace here so that you can see my configuration

alang@ubuntu-alang:~$ fluentd -c /home/alang/fluentd/fluent.conf  &
[1] 14633
alang@ubuntu-alang:~$ 2016-09-13 17:09:22 +0300 [info]: reading config file path="/home/alang/fluentd/fluent.conf"
2016-09-13 17:09:22 +0300 [info]: starting fluentd-0.14.6
2016-09-13 17:09:22 +0300 [info]: spawn command to main: /usr/bin/ruby2.3 -Eascii-8bit:ascii-8bit /usr/local/bin/fluentd -c /home/alang/fluentd/fluent.conf --under-supervisor
2016-09-13 17:09:22 +0300 [info]: reading config file path="/home/alang/fluentd/fluent.conf"
2016-09-13 17:09:22 +0300 [info]: starting fluentd-0.14.6 without supervision
2016-09-13 17:09:22 +0300 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2016-09-13 17:09:22 +0300 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-09-13 17:09:22 +0300 [info]: gem 'fluent-plugin-bigquery' version '0.2.16'
2016-09-13 17:09:22 +0300 [info]: gem 'fluent-plugin-buffer-lightening' version '0.0.2'
2016-09-13 17:09:22 +0300 [info]: gem 'fluentd' version '0.14.6'
2016-09-13 17:09:22 +0300 [info]: adding match pattern="cda.all" type="bigquery"
2016-09-13 17:09:22 +0300 [info]: adding source type="tail"
2016-09-13 17:09:22 +0300 [info]: using configuration file: <ROOT>
  <source>
    @type tail
    format /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\,\d{3}) (?<level>(DEBUG|TRACE|ERROR| WARN| INFO)) (?<source>(\w+:\d+)) - (?<message>(.+$))/
    path "/usr/local/reports/root/logs/All.log"
    pos_file "/usr/local/reports/root/logs/cda-all.pos"
    read_from_head true
    tag "cda.all"
    refresh_interval 5
    <parse>
      @type /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\,\d{3}) (?<level>(DEBUG|TRACE|ERROR| WARN| INFO)) (?<source>(\w+:\d+)) - (?<message>(.+$))/
    </parse>
  </source>
  <match cda.all>
    @type bigquery
    method "insert"
    auth_method "json_key"
    json_key /home/alang/.keys/xxxxx-xxxxxxxxxxxxx.json
    email "xxxxxxxxxxxxx...@developer.gserviceaccount.com"
    buffer_type "file"
    buffer_path /usr/local/reports/root/logs/buffer/bq
    project "xxxxx-analytics"
    dataset "analytics"
    tables "log_aggregator"
    field_string "time, level, source, message"
    <buffer time>
      flush_mode lazy
      retry_type exponential_backoff
      @type file
      path /usr/local/reports/root/logs/buffer/bq
      timekey 86400
    </buffer>
  </match>
</ROOT>
2016-09-13 17:09:22 +0300 [error]: unexpected error error_class=ArgumentError error="wrong number of arguments (given 1, expected 0)"
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/output.rb:333:in `start'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/compat/output.rb:655:in `start'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluent-plugin-bigquery-0.2.16/lib/fluent/plugin/out_bigquery.rb:233:in `start'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/compat/call_super_mixin.rb:42:in `start'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:138:in `block in start'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:127:in `block (2 levels) in lifecycle'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:126:in `each'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:126:in `block in lifecycle'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:113:in `each'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:113:in `lifecycle'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:137:in `start'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/engine.rb:211:in `start'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/engine.rb:175:in `run'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/supervisor.rb:677:in `run_engine'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/supervisor.rb:436:in `block in run_worker'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/supervisor.rb:606:in `main_process'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/supervisor.rb:431:in `run_worker'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/command/fluentd.rb:271:in `<top (required)>'
  2016-09-13 17:09:22 +0300 [error]: /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2016-09-13 17:09:22 +0300 [error]: /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/bin/fluentd:5:in `<top (required)>'
  2016-09-13 17:09:22 +0300 [error]: /usr/local/bin/fluentd:23:in `load'
  2016-09-13 17:09:22 +0300 [error]: /usr/local/bin/fluentd:23:in `<main>'

This is the fluent.conf file

@type tail

format /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\,\d{3}) (?<level>(DEBUG|TRACE|ERROR| WARN| INFO)) (?<source>(\w+:\d+)) - (?<message>(.+$))/

path /usr/local/reports/root/logs/All.log

pos_file /usr/local/reports/root/logs/cda-all.pos

read_from_head true

tag cda.all

refresh_interval 15

</source>

# forwarding to bigquery plugin

@type bigquery

method insert

auth_method json_key

email "xxxxxxxxxxxxx...@developer.gserviceaccount.com"

json_key /home/alang/.keys/xxxxx-xxxxxxxxxxxxx.json

buffer_type file

buffer_path /usr/local/reports/root/logs/buffer/bq

project cellebrite-analytics

dataset analytics

tables log_aggregator

time_format %s

time_field time

field_string level, source, message

</match>

Alan

Satoshi Tagomori

unread,

Sep 14, 2016, 10:15:30 PM9/14/16

to flu...@googlegroups.com

Hi Alan,

I checked the code of fluent-plugin-bigquery, and confirmed that it must be fixed at some points

to run with Fluentd v0.14.

Please wait update releases, or try to write/send patch to that plugin :)

Moris.

xxxxxxxxxxxxx.json
    email "xxxxxxxxxxxxxxxxxxxxxxxxxx@developer.gserviceaccount.com"
    buffer_type "file"
    buffer_path /usr/local/reports/root/logs/buffer/bq
    project "xxxxx-analytics"
    dataset "analytics"
    tables "log_aggregator"
    field_string "time, level, source, message"
    <buffer time>
      flush_mode lazy
      retry_type exponential_backoff
      @type file
      path /usr/local/reports/root/logs/buffer/bq
      timekey 86400
    </buffer>
  </match>
</ROOT>
2016-09-13 17:09:22 +0300 [error]: unexpected error error_class=ArgumentError error="wrong number of arguments (given 1, expected 0)"
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/plugin/output.rb:333:in `start'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/compat/output.rb:655:in `start'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluent-plugin-bigquery-0.2.16/lib/fluent/plugin/out_bigquery.rb:233:in `start'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/compat/call_super_mixin.rb:42:in `start'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:138:in `block in start'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:127:in `block (2 levels) in lifecycle'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:126:in `each'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:126:in `block in lifecycle'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:113:in `each'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:113:in `lifecycle'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:137:in `start'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/engine.rb:211:in `start'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/engine.rb:175:in `run'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/supervisor.rb:677:in `run_engine'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/supervisor.rb:436:in `block in run_worker'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/supervisor.rb:606:in `main_process'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/supervisor.rb:431:in `run_worker'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/lib/fluent/command/fluentd.rb:271:in `<top (required)>'
  2016-09-13 17:09:22 +0300 [error]: /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2016-09-13 17:09:22 +0300 [error]: /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2016-09-13 17:09:22 +0300 [error]: /var/lib/gems/2.3.0/gems/fluentd-0.14.6/bin/fluentd:5:in `<top (required)>'
  2016-09-13 17:09:22 +0300 [error]: /usr/local/bin/fluentd:23:in `load'
  2016-09-13 17:09:22 +0300 [error]: /usr/local/bin/fluentd:23:in `<main>'

email "xxxxxxxxxxxxxxxxxxxxxxxxxx@developer.gserviceaccount.com"

json_key /home/alang/.keys/xxxxx-xxxxxxxxxxxxx.json

buffer_type file
buffer_path /usr/local/reports/root/logs/buffer/bq

project cellebrite-analytics
dataset analytics
tables log_aggregator

time_format %s
time_field time

field_string level, source, message

</match>

Alan

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

田籠聡 (TAGOMORI Satoshi)
Mail: tago...@gmail.com

Alan Green

unread,

Sep 15, 2016, 3:39:37 AM9/15/16

to Fluentd Google Group

Thanks very much.

Reply all

Reply to author

Forward

0 new messages