The truncate table vs update schema is not clear to me. When a table is truncated does the schema no longer matter? Why can’t I update the schema of a table I am truncating (as specified
here)?
I ran into an issue where my pipelines can fail when the schema is updated on a sink that is set to both truncate and update the schema. Bigquery is complaining that CDAP is giving it a new schema. How should this be dealt with?
From the CDAP logs:
{
"code" : 400,
"errors" : [ {
"domain" : "global",
"message" : "Provided Schema does not match Table bianalyticis:yardi_sql_server.unit. Field RentType has changed type from STRING to INTEGER",
"reason" : "invalid"
} ],
"message" : "Provided Schema does not match Table bianalyticis:yardi_sql_server.unit. Field RentType has changed type from STRING to INTEGER",
"status" : "INVALID_ARGUMENT"
}
Ironically, when I rerun the same pipeline it works the second time. Makes me think that although BQ rejected the data because of a schema update, it still updated the schema silently 🤷🏻♀️?