Data Drift for SQL Change Tracking

[Srini Sankar]

Dear Members,

I would like to apply a Data Drift rule for a pipeline that picks deltas based on SQL Change Tracking. 

I am curious to understand how would the following DRIFT functions work,

drift:names(<field path>, <ignoreWhenMissing: true | false>)

drift:order(<field path>, <ignoreWhenMissing: true | false>)

drift:size(<field path>, <ignoreWhenMissing: true | false>)

drift:type(<field path>, <ignoreWhenMissing: true | false>)

My pipeline is a batch processing that runs every 10 minutes. Would the above functions look for changes only within a same batch or across batches.

Documentation says "Drift functions are checked for two subsequent JSON records". But I would like to confirm whether the behavior is same for all functions listed above or it only applies to drift:names() and drift:order() functions.

Reason - I pull from an application's backend DB and schema changes happen very rarely and would like to understand whether I can apply Drift functions on SQL Change Tracking on a batch mode.

Thanks,

Srini