Ordering in Debezium

[Shuguang Xiang]

Hi team,

From Debezium docs, we find that binlog event from Debezium can be pushed into Kafka topics in two ways:

1. Round robin
2. Partition by primary key (which is primary key),  ensuring ordering for same primary key within a topic partition

Is option 2 default behaviour in Debezium?

And is primary key configurable? For example, can I choose a column as a partition key in Kafka when pushing message to kafka topic?

Best regards,

Shuguang

[Chris Cranford]

Hi Shuguang,

Yes, by default Debezium emits messages based on the primary key columns of a table as the message key.  This means for a single partitioned topic, not only will the order of operations against the same key be preserved, but all operations against that table will be emitted in order.  If your topic uses a multiple partition scheme, then the order is only preserved for messages with the same primary key / message key; where-as the order for rows with differing primary keys won't be possible.  If you want to alter the message key, this can be done by applying a custom SMT that modifies the emitted message from Debezium to fit whatever business use case you need.  For details on routing by topic or key manipulation, see https://debezium.io/documentation/reference/1.2/configuration/topic-routing.html.

Thanks,
Chris C.

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to the processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email Please notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/8313c6d7-5fcc-48fd-b41a-18ee162abe7bn%40googlegroups.com.

[Shuguang Xiang]

Thanks for explanation, Chris

On 28 Jul 2020, at 12:21 AM, Chris Cranford <cran...@gmail.com> wrote:

Unsubscribe

It appears that you have subscribed to commercial messages from this sender. To stop receiving such messages from this sender, please unsubscribe

To view this discussion on the web visit http://secure-web.cisco.com/1bx-gpwCU27MrKVuNv_GI526krorA26LA7QZzrq1JX4u2ioCOXmpRXLx9k3jAxZyr3v_UqW3QCXz5pRSBuiyZ2xC2h0TJGrBlsrfb7sG7phcHcind91QPXii9sQPyeB2kJgcJPLCt2Vy_yRjXwEHhGaGcvU12EY_4Ok0Aw6Gdb_gss1erRgIxOujvJk6BUsHJufNgptNg_TNeRT0RZwicjmuErIIKRaNsnfuUkiO_oRcjasttsS81Ge6ZqhBbI0qZtyIs9fEKz6D23Ql4KmYHNKEmMTY1loOBo4D3WjGKgjx1q60iwfk_divnFgRnKWwbI5EkBe2qPjHsUV3ObmAIi5vWl6XGkZWU1S2t9eWEgdw41HyjFeDli2_GuJxUnuOmBd1ii2Lta3SQH_VGx_hZtdH_4AMa_pFxKqC0nTiNrgfvKwirm8m4tp-DhN6sE0E9SwO8GOznu1t1NQ8x8bfFiLGdDa5w7CWJQlKtvpDerD0EastYIRYa10p5FYqoWf2FB4CkBhTQ7zstlMNxRjB7xpKzN9kg1tq459UOK3W-jgM/l99%3Ahttps%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fdebezium%2F8313c6d7-5fcc-48fd-b41a-18ee162abe7bn%2540googlegroups.come.

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.

To view this discussion on the web visit http://secure-web.cisco.com/1hYRBTTnRAYlZj0guiLzK9VxAA9VDxXox4M6BbqGOpGgWs8lLRN-sBiLj-djpwsVDERkaUTZQCSlD7smMv_XfnIKlp3ZdhejLmp4dRP5pTm9X_rwy-jmtDR3cIjjucpN_S8tlfWeW3r4CSTSrk71MtOJe8T-tTngqOdakcYDPTI_Ru63yJF2dydBEyF5AeEOsL9ClEYCIER8JJlUjOFCRrots7qPB27vDf8oKMiNvV6ndTxACg2FAfKCdGZkq9XGVfdG9j5JZp64iUHN3WGnpEuycWdB0sOS1iyhjTCRIEO94NAbpk2Hv0BzzvFa_MCsxxuZy29cGLriQh6uU6XrlEkNlnDDCI4aBQm6n3AGZrd3lYFbfeq2tgVXUlasq9WXbFpdeTCGLr7T832bsiKjEA-ttJGZS5g3UDDRiPEWhEjwp_9iuytM51AFa3HiioXBGUf-fp5Q347MJ9nzFx5Wgt_emB6XFcrSUQql3Gzeq7rowsqLUMvC4cEy3FQK5VdQW5Ov9IrlaFtiBn6x5q3aCJBqDMS-nfQzlFGIJ7Mtd-E4/l91%3Ahttps%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fdebezium%2Fc955fdaa-2d06-ea94-317c-e7dd1717cf9d%2540gmail.come.

[Gunnar Morling]

What Chris said :)

Customizing the key does not require an SMT, though; you can use the "message.key.columns" option of the relational Debezium connectors (e.g. see here for Postgres: https://debezium.io/documentation/reference/connectors/postgresql.html#postgresql-property-message-key-columns) to control which columns should make up the message key and thus will impact the partition assignment.

--Gunnar