Fallback to Secondary Plugin

[Sanket Gupta]

Hi

i am trying to write a buffered plugin that uploads a chunk of a buffer to a third party REST service in chunks of 500 KB.

Although this part is working fine, i would like to add support for fallback in case the third party service is down.

basically, whenenver my http command fails, i want to send the buffer to a different plugin which will then take corrective action (For eg: Upload the file to s3).

To achieve this should i iterate the records in the buffer and re-emit them or is there a better way to  pass on the whole chunk to the fallback plugin?

Regards

Sanket

[Mr. Fiber]

> To achieve this should i iterate the records in the buffer

There is no re-iterate mechanizm because events are serialized before flush.

Secondary receives failed chunk but secondary can't know 'how to parse received chunk'.

So fluentd calls secondary's write with failed chunk immediately. No format or format_stream call.

This is why fluentd recommends file or similar plugin to backup failed chunks.

> is there a better way to  pass on the whole chunk to the fallback plugin?

s3 plugin seems to work in secondary without additional work but I don't test it...

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sanket Gupta]

Hi

Thanks for this.Secondary output seems tailor made for my use case :) . I will try secondary output with s3 and see if it gives the desired results.

Regards

Sanket

[Sanket Gupta]

it worked for my case as i was uploading raw json chunks, scondary output got the same raw data

[Mr. Fiber]

Good to hear that :)

[Sanket Gupta]

:) 

As usual just one thing to say . Fluentd is just brilliant.Seems to have all the features that anybody may need! Keep up the great work

[Yasin Amadmia]

Hi,

I want to understand the fallback behaviour. 

Say a plugin fails over, exhausts its retry limits and secondary kicks in.

What I want to know if each 'flush' would now go through the same route ? i.e. at each flush the first plugin tried and if it fails then secondary used or now that it has failed over, secondary will be used all the time ? If later, when will fall back from secondary to main happen ?

Thanks.