I'm trying to ingest one table from a linked server in a local server that points to another remote database in the Azure environment.
The target is a big query in the google cloud environment.
For a small table, it works fine, preview and running.
For my biggest table with 160 million rows, it is taking forever till the pipeline crash.
I'm using a multi-table JDBC component to get data from the source and Big Query to the destination, no transformation, simple data transfer pipeline.
Does anybody face something similar?
I believe the reason is that Streamsets cannot get the source(azure) table metadata to identify which column is the primary key to defining an offset column, which I have tried to override with no success.
I'll appreciate any thoughts or advice.