Dynamically BigQuery schema creation
1,387 views
Skip to first unread message
Sergey Petrovsky
Jan 19, 2021, 4:52:35 AM1/19/21
to Google Cloud Developers
Hello.
I wrote the transormer than maps Avro schema (placed in .avsc file) to the BigQuery schema.
I have use the com.google.api.services.bigquery.model.TableSchema class. It is sort of flexible to fill it dynamically due to recursion walking through .avsc file. But I end up with difficulties that I can't use TableSchema for creating the BQ schema using BQ java SDK.
SDK push me to use the com.google.cloud.bigquery.Schema that not so flexible to fill it dynamically.
Can someone, please, clarify how I can create BQ schema dynamically and create an empty BQ native table using java SDK?
Manuel Alaman Escolano
Jan 21, 2021, 11:24:12 AM1/21/21
to Google Cloud Developers
Hi,
Regarding your question on how to create the BQ schema dynamically, I would suggest that you checked [1] to have a better understading on how to use the Schema autodetection. According to [2], "Schema auto-detection is not used with Avro files, Parquet files, ORC files, Firestore export files, or Datastore export files. When you load these files into BigQuery, the table schema is automatically retrieved from the self-describing source data". In case you want to insert JSON or CSV data, you should use the .setAutodetect() flag to true. Please refer to [2] to see how it is done.
In order to create an empty table without a schema definition, please refer to [3]. In case you want to create a table with a schema definition, you should specify the schema using one of the proposed options. Please take a look at the documentation on how to define a schema in BQ [5].
Additionally, check the specific documentation on how to load avro data into a new table [6].
________________________________
[1] -
[2] -
[3] -
[4] -
[5] -
[6] -
Reply all
Reply to author
Forward
0 new messages