Google Cloud KMS Best Practice with BigQuery
Kishan Kumar
I need to Encrypt the Sensitive fields in the Bq Table but my Loading Is Done through the Dataflow. I thought of 3 Different way to Use it.
1. Encrypt the whole Table using Customer Managed Key and Make 3 Views on Different Classifications and provide Service account to Users to access the View and Provide that Service account role as Decrypter in KMS and Dataflow Service Account as Encrypter Load the Table. (Problem We do not have View Level Access so that views Required to Maintain in Different Datasets which makes our job more Difficult)
2. Encrypt the Fields Using The API call in Dataflow While Loading and Make a UDF function to Decrypt that Colum Data at Runtime in Bq Using Service Account.
Example Id Fields are Encrypted Using API call in Dataflow And we defined a UDF function in Bq to Decrypt it but only those can decrypt that Data who have access in KMS else it will throw an Exception
In this way, we keep a Single Table Open to All Users but Only Authenticated User can only See the that.
Problem: (Continuous Call of API at Runtime which makes our quota Exhausted and Cost is Another Matter)
3.Maintaining Different tables in different datasets which
Problem: (Maintenance and Making Data in Sink and Join at Run Time in BQ)
a. Encrypted Tables with Sensitive Field
b. Non-Encrypted Table with Non-Sensitive Fields.
The Above are My Approach and Use case Is Anyone able to help me to see what to Use and Why its better than others.
