Issues updating schema (BigQuery)

[Noah Sragow]

The truncate table vs update schema is not clear to me. When a table is truncated does the schema no longer matter? Why can’t I update the schema of a table I am truncating (as specified here)?

I ran into an issue where my pipelines can fail when the schema is updated on a sink that is set to both truncate and update the schema. Bigquery is complaining that CDAP is giving it a new schema. How should this be dealt with?

From the CDAP logs:

{
  "code" : 400,
  "errors" : [ {
    "domain" : "global",
    "message" : "Provided Schema does not match Table bianalyticis:yardi_sql_server.unit. Field RentType has changed type from STRING to INTEGER",
    "reason" : "invalid"
  } ],
  "message" : "Provided Schema does not match Table bianalyticis:yardi_sql_server.unit. Field RentType has changed type from STRING to INTEGER",
  "status" : "INVALID_ARGUMENT"
}

Ironically, when I rerun the same pipeline it works the second time. Makes me think that although BQ rejected the data because of a schema update, it still updated the schema silently 🤷🏻‍♀️?

All the best,

Noah Sragow Software Engineer M: 323-992-9636 noa...@westlandreg.com Westland Real Estate Group 520 W. Willow, Long Beach, CA 90806 www.westlandreg.com

[Albert Shau]

Hi Noah,

Which version of the BigQuery sink are you using? If it is an older version, can you try installing the latest version from the Hub? I believe this was fixed as part of https://cdap.atlassian.net/browse/CDAP-16879.

Thanks,

Albert

--
You received this message because you are subscribed to the Google Groups "CDAP User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdap-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cdap-user/CACP_uRCwF1qTesBsdF9DCDBRhxCfrUYjCb3Z0FR_72hpOOJ_Lg%40mail.gmail.com.