quick question: how to refer to the log as a whole in the match part?

[Fere]

Hi,

We have diff Json logs coming through from the source section.  We want to send all of them (varied) as a Json blob to a a database table expecting just one field (to hold the Json blob). 

The output plugin I am using is BigQuery which has a param called field_string.  I have to pass the name of the field from the input log.  What field name shoudl I use so that it passes the entire log as one field?

Thanks,

Fere

[Mr. Fiber]

> What field name shoudl I use so that it passes the entire log as one field?

You can use any name.

Using filter to set entire log into one field and refer it in output plugin.

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Fere]

Thanks for quick response.

I read through the online doc on filter and all I could see at the end was a reference to formatted_record which I tried even though i thought that was not really what you meant.  and, not surprsingly, it did not work. I am still not clear on how to make a field that is basically the original raw json log.  Appreciate your help.

Here is my conf:

<filter kafka.consumer.mytopic>

  @type record_transformer

  <record>

     raw_log   formatted_record

  </record>

</filter>

<match kafka.consumer.mytopic>

    @type bigquery

    auth_method json_key

    json_key /fluentd/etc/LL-POC-9081311ba6a0.json

    project ll-poc

    dataset SL_POC

    table   sansar_logs_v1

    auto_create_table true

   field_string raw_log

   buffer_queue_limit 10240        

   num_threads 16

   buffer_type file

   buffer_path /var/log/buffer/bq

</match>

[Fere]

Would really appreciate any help with this.

I am stuck, because form the online docs, I can't figure out how to refer/create a field that contains the entire log.

Any suggestions?

[Mr. Fiber]

You can access event member, tag, time and record, in ${} with "enable_ruby true".

In "enable_ruby true", you can write any ruby code.

[Fere]

Thx.  the problem is I want to access the whole log as one field (one json blob) rather than picking individual event fields/members.  Can that be done?

[Mr. Fiber]

Try ${record}.