Will Debezium built-in SMT "ExtractNewRecordState" support excluding unchanged fields?

[Brandon Hsu]

Hi, All,

As far as I know, Debezium provides a built-in SMT "ExtractNewRecordState" to extract and flatten "payload.after" content.

https://debezium.io/documentation/reference/stable/transformations/event-flattening.html

But sometimes it is useful to get only changed columns and optionally primary/unique key columns in a flattened record for further data handling.

I think it might be great if "ExtractNewRecordState" can optionally compare "payload.before" / "payload.after" fields/values and only keeping changed after-state fields to compose the result flattened record.

And such feature may be controlled by providing new "ExtractNewRecordState" configurations to turn on/off such feature

 - "drop.unchanged.fields" : to keep only changed fields in the resulting flattened record

 - "include.primary.keys" : to optionally keep primary/unique key fields in the resulting record no matter the previous setting is present or not.

Is it possible for Debezium community to implement such feature in the future?

Best Regards.

----------------------------------------------------------------------------------------------------------------------------------------------------
本信件(包括任何附件)為機密信件。若您並非指定收信人，嚴禁複製、傳送、公開或使用其所載的全部或部份資料，任何對於本信件未經授權之使用或散播可能帶有法律責任。
若您偶然獲得此郵件，請回信或立即以電話通知寄件人並且從您的電腦系統中刪除本信件及所有附件，謝謝。
The information in this e-mail may be confidential; it is intended for use solely by the individual or entity named as the recipient hereof. Disclosure, copying, distribution, or use of the contents of this e-mail by persons other than the intended recipient may violate applicable laws and if you have received this e-mail in error, please delete the original message and notify us by collect call immediately. Thank you.

----------------------------------------------------------------------------------------------------------------------------------------------------

[Chris Cranford]

Hi Brandon -

Great ideas. 

So I don't particularly have any issues adding these toggles, as long as they're opt-in.  In many cases, ExtractNewRecordState is paired with a JDBC sink connector in a DB replication pipeline and these would most definitely not be compatible with that scenario.  That said, there may be other SMT or CDC pipelines where these might actually be quite useful and it makes logical sense to include those as a part of that SMT I think.  I would suggest raising a Jira enhancement issue at https://issues.redhat.com/projects/DBZ and if you or someone else from the community wishes to contribute the work, we'd be welcomed to review it!

Thanks,
CC

---------------------------------------------------------------------------------------------------------------------------------------------------- --
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/CA%2BJYH_1tSP4AnAtzuNmiyuJWMnpjcroDoeJAZbvYp35vVt2fzA%40mail.gmail.com.

[CCHsu]

Hi, Chris,

Thanks for your feedback.  😄

I have raised a Jira enhancement here.

https://issues.redhat.com/browse/DBZ-5283

Hope this is not hard to implement.  😊

Chris Cranford 在 2022年6月21日 星期二晚上8:09:22 [UTC+8] 的信中寫道：