Multiple outputs - S3 and BigQuery

[Ashish Tilara]

Based on suggested found https://github.com/fluent/fluentd/issues/1110, with multiple workers,

<system>

    workers 2

</system>

<match debug>

    @id s3debug 

    @type copy 

    <store>

        @id bqdebug

        @type bigquery 

#------> file buffer

    </store>

    <store>

        @id s3debug

        @type s3 

#------> file buffer

    </store>

</match>

I ideally want to use file buffer with it, but somehow when I put following block for S3 store,

      <buffer tag,time>

        @type "file"

        path "/tmp/s3.*.buffer"

        timekey 15m

        timekey_use_utc true

        chunk_limit_size 1m

      </buffer>

does not work. :'( 

2021-04-30 04:13:02 +0000 [error]: #0 [s3debug] file buffer with multi workers should be configured to use directory 'path', or system root_dir and plugin id

2021-04-30 04:13:02 +0000 [error]: #0 config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Plugin 'file' does not support multi workers configuration (Fluent::Plugin::FileBuffer)"

I would appreciate any help.

[Mr. Fiber]

See https://docs.fluentd.org/deployment/multi-process-workers#root_dir-id-parameter

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluentd/1a8f7b60-6ab8-4bc4-be68-7ff4841339dan%40googlegroups.com.

[Ashish Tilara]

Thank you , following structure worked.

<system>

    workers 2

    root_dir /fluentd

</system>

<match debug>

    @id s3debug 

    @type copy 

    <store>

        @id bqdebug

        @type bigquery 

        <buffer>

            @type file

            flush_interval 1m

        </buffer>

        #.... other params

    </store>

    <store>

        @id s3debug

        @type s3 

        <buffer tag,time>

    @type file

    timekey_use_utc true

    timekey 15m

        </buffer>

        #.... other params

    </store>

</match>