Data Drift for SQL Change Tracking

Skip to first unread message

Srini Sankar

Jul 9, 2021, 9:51:07 AM7/9/21
to sdc-user
Dear Members,

I would like to apply a Data Drift rule for a pipeline that picks deltas based on SQL Change Tracking. 

I am curious to understand how would the following DRIFT functions work,

drift:names(<field path>, <ignoreWhenMissing: true | false>)
drift:order(<field path>, <ignoreWhenMissing: true | false>)
drift:size(<field path>, <ignoreWhenMissing: true | false>)
drift:type(<field path>, <ignoreWhenMissing: true | false>)

My pipeline is a batch processing that runs every 10 minutes. Would the above functions look for changes only within a same batch or across batches.

Documentation says "Drift functions are checked for two subsequent JSON records". But I would like to confirm whether the behavior is same for all functions listed above or it only applies to drift:names() and drift:order() functions.

Reason - I pull from an application's backend DB and schema changes happen very rarely and would like to understand whether I can apply Drift functions on SQL Change Tracking on a batch mode.


Reply all
Reply to author
0 new messages