How to consume from beginning of topic using MirrorMaker

513 views
Skip to first unread message

Stuart Wong

unread,
Sep 29, 2015, 3:56:05 PM9/29/15
to Confluent Platform
I'm hoping someone can help me as I've about hit a wall. I've got MirrorMaker running successfully. Any new message coming into topic `_schema` in DC-A is successfully replicated to `_schema` in DC-B. However, I would like to replicate the messages from the very start of the topic but have been unable to get this to work.

I've removed the MirrorMaker consumer from Zookeeper, and restarted MirrorMaker but that did not work (not that I thought it would).

Any ideas or insight?

Thanks,
Stuart.

Geoff Anderson

unread,
Sep 29, 2015, 4:54:27 PM9/29/15
to confluent...@googlegroups.com
Hi Stuart,

You'll probably need to make sure that the consumer config for your MirrorMaker instance includes the setting
auto.offset.reset=smallest

This defaults to largest, so if it is not set, your consumer will consume from the end of the topic by default.

Hope this helps,
Geoff



--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/c768123c-ce9b-4be4-a732-1f6667cafeb6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Carrington Wong

unread,
Sep 29, 2015, 6:02:04 PM9/29/15
to confluent...@googlegroups.com
I'll try tasty and see. Thanks.
You received this message because you are subscribed to a topic in the Google Groups "Confluent Platform" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/confluent-platform/w8K2HcqXF0Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/CAM%3DixS%2BeiVoR4aSR%3Di%3D%2BTC3_760rsxdCzKROT10QJ2vjP1QttQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.


--

Stuart Wong

unread,
Sep 30, 2015, 1:45:06 PM9/30/15
to Confluent Platform
This worked. Thanks!


On Tuesday, September 29, 2015 at 3:54:27 PM UTC-5, Geoffrey Anderson wrote:
Hi Stuart,

You'll probably need to make sure that the consumer config for your MirrorMaker instance includes the setting
auto.offset.reset=smallest

This defaults to largest, so if it is not set, your consumer will consume from the end of the topic by default.

Hope this helps,
Geoff


On Tue, Sep 29, 2015 at 12:56 PM, Stuart Wong <cgs....@gmail.com> wrote:
I'm hoping someone can help me as I've about hit a wall. I've got MirrorMaker running successfully. Any new message coming into topic `_schema` in DC-A is successfully replicated to `_schema` in DC-B. However, I would like to replicate the messages from the very start of the topic but have been unable to get this to work.

I've removed the MirrorMaker consumer from Zookeeper, and restarted MirrorMaker but that did not work (not that I thought it would).

Any ideas or insight?

Thanks,
Stuart.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platform+unsub...@googlegroups.com.

Geoff Anderson

unread,
Sep 30, 2015, 2:09:24 PM9/30/15
to confluent...@googlegroups.com
Glad to hear it Stuart.

Cheers,
Geoff

On Wed, Sep 30, 2015 at 10:45 AM, Stuart Wong <cgs....@gmail.com> wrote:
This worked. Thanks!

On Tuesday, September 29, 2015 at 3:54:27 PM UTC-5, Geoffrey Anderson wrote:
Hi Stuart,

You'll probably need to make sure that the consumer config for your MirrorMaker instance includes the setting
auto.offset.reset=smallest

This defaults to largest, so if it is not set, your consumer will consume from the end of the topic by default.

Hope this helps,
Geoff


On Tue, Sep 29, 2015 at 12:56 PM, Stuart Wong <cgs....@gmail.com> wrote:
I'm hoping someone can help me as I've about hit a wall. I've got MirrorMaker running successfully. Any new message coming into topic `_schema` in DC-A is successfully replicated to `_schema` in DC-B. However, I would like to replicate the messages from the very start of the topic but have been unable to get this to work.

I've removed the MirrorMaker consumer from Zookeeper, and restarted MirrorMaker but that did not work (not that I thought it would).

Any ideas or insight?

Thanks,
Stuart.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.

To post to this group, send email to confluent...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages