Needs help with td-agent multi kafka-input/file-output

[Thang Pham]

Hi all,

I'm new to td-agent and trying to use it to export Kafka log topics to files (time slice feature is exactly what I need). I'm using td-agent 2.3.6 on Debian 8. My configuration is like this:

$ cat /etc/td-agent/td-agent.conf
@include conf.d/*.conf

$ cat /etc/td-agent/conf.d/out_topic1.conf
<source>
  @type kafka_group

  brokers sv1:9092,sv2:9092,sv3:9092
  consumer_group td-agent-file-out
  topics topic1
  format text

  start_from_beginning false
</source>

<match ugh_omnibox>
  @type file
  path /var/kafka/topic1/topic1
  symlink_path /var/kafka/topic1/current
  buffer_path /dev/shm/topic1.td-agent.buffer
  @include out_file.inc
</match>

$ cat /etc/td-agent/conf.d/out_file.inc
append true
format single_value
compress gzip
time_slice_format %Y%m%d
buffer_type file
slow_flush_log_threshold 50.0

It works totally fine when I have only 1 topic like this. But when I added 1 more file out_topic2.conf for another topic, it stopped working properly. There's only 1 in 2 topics being written.

Could somebody please help to show me where I'm wrong?

Thank you very much and have a nice week,
--
Thang Pham

[Mr. Fiber]

Does out_topic2.conf change only topics parameter?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Thang Pham]

Hi Masahiro,

I basically did 's/topic1/topic2/' on all the config file.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

[Thang Pham]

One more important thing, I keep receiving a lot of these in td-agent log file:

2018-04-10 10:04:47 +0700 [error]: unexpected error during consuming events from kafka. Re-fetch events. error="Kafka::NoPartitionsAssignedError"
2018-04-10 10:04:47 +0700 [error]: unexpected error during consuming events from kafka. Re-fetch events. error="Kafka::NoPartitionsAssignedError"
  2018-04-10 10:04:47 +0700 [error]: suppressed same stacktrace
  2018-04-10 10:04:47 +0700 [error]: suppressed same stacktrace

[Thang Pham]

Now I use different consumer group for kafka and the problem is gone.

Thank you Masahiro for your hint.

On Monday, April 9, 2018 at 6:19:09 PM UTC+7, repeatedly wrote:

Does out_topic2.conf change only topics parameter?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.