Will Debezium built-in SMT "ExtractNewRecordState" support excluding unchanged fields?

Skip to first unread message

Brandon Hsu

Jun 16, 2022, 11:47:13 AMJun 16
to debezium
Hi, All,

As far as I know, Debezium provides a built-in SMT "ExtractNewRecordState" to extract and flatten "payload.after" content.

But sometimes it is useful to get only changed columns and optionally primary/unique key columns in a flattened record for further data handling.

I think it might be great if "ExtractNewRecordState" can optionally compare "payload.before" / "payload.after" fields/values and only keeping changed after-state fields to compose the result flattened record.

And such feature may be controlled by providing new "ExtractNewRecordState" configurations to turn on/off such feature
 - "drop.unchanged.fields" : to keep only changed fields in the resulting flattened record
 - "include.primary.keys" : to optionally keep primary/unique key fields in the resulting record no matter the previous setting is present or not.

Is it possible for Debezium community to implement such feature in the future?

Best Regards.

The information in this e-mail may be confidential; it is intended for use solely by the individual or entity named as the recipient hereof. Disclosure, copying, distribution, or use of the contents of this e-mail by persons other than the intended recipient may violate applicable laws and if you have received this e-mail in error, please delete the original message and notify us by collect call immediately. Thank you.


Chris Cranford

Jun 21, 2022, 8:09:22 AMJun 21
to debe...@googlegroups.com, Brandon Hsu
Hi Brandon -

Great ideas. 

So I don't particularly have any issues adding these toggles, as long as they're opt-in.  In many cases, ExtractNewRecordState is paired with a JDBC sink connector in a DB replication pipeline and these would most definitely not be compatible with that scenario.  That said, there may be other SMT or CDC pipelines where these might actually be quite useful and it makes logical sense to include those as a part of that SMT I think.  I would suggest raising a Jira enhancement issue at
https://issues.redhat.com/projects/DBZ and if you or someone else from the community wishes to contribute the work, we'd be welcomed to review it!

---------------------------------------------------------------------------------------------------------------------------------------------------- --
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/CA%2BJYH_1tSP4AnAtzuNmiyuJWMnpjcroDoeJAZbvYp35vVt2fzA%40mail.gmail.com.


Jun 21, 2022, 10:12:45 AMJun 21
to debezium
Hi, Chris,

Thanks for your feedback.  😄

I have raised a Jira enhancement here.

Hope this is not hard to implement.  😊

Chris Cranford 在 2022年6月21日 星期二晚上8:09:22 [UTC+8] 的信中寫道:
Reply all
Reply to author
0 new messages