CloudSQL PostgreSQL change data capture (CDC)

[Gordon Baird]

Does GCP CloudSQL support CDC or any other type of record mutation/change event triggering?  If not, is this in the roadmap?     If not, is there any other way to trigger events (functions, pub/sub) by changes to records in the database?  If not CloudSQL, do any other GCP ACID compliant dbases support them?   Thanks

[Geert Van Landeghem]

CloudSQL PostgreSQL does not support CDC at this time, nor is it known to be on the roadmap to ever be enabled.

You can host your own PostgreSQL database (eg using Kubernetes and Docker) and use CDC there.

CloudSQL Sql Server also does not support CDC at this moment.

Currently,CloudSQL MySQL is the only database that offers CDC using its binlog.

On Fri, Sep 18, 2020 at 2:52 PM Gordon Baird <gor...@bairdpartnersltd.com> wrote:

Does GCP CloudSQL support CDC or any other type of record mutation/change event triggering?  If not, is this in the roadmap?     If not, is there any other way to trigger events (functions, pub/sub) by changes to records in the database?  If not CloudSQL, do any other GCP ACID compliant dbases support them?   Thanks

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/d79e20b0-234e-4fb6-8e7b-b10139874985n%40googlegroups.com.

--

kind regards,

--

Geert Van Landeghem

Training & Operations & Co-Founder

T R E E 5 3

Connect. Transform. Grow. 

T  +32 (0) 477 75 95 33

E  geert.van...@tree53.com

[Gordon Baird]

Any thoughts on how then to design a trigger or event that kicks of down stream services that is key'd off a write to a SQL ACID dbase?  other designs include periodic polling or cloud functions , but ideally looking for something more native or lower level in one of the cloud housed dbase services.. 

[Amit Sinha]

Hello,

From this documentation, CDC features are not yet available. However, I have found some feature requests related to this [1,2]. If you want any specific feature that could be helpful, you can create another feature request here.

Workaround for now, manually deploy SQL cluster on GCE VMs; you could deploy a self-managed SQL cluster on GCE VMs and monitor the activity through there. This would require additional effort for manual engineering and operations unfortunately. 

Another alternative would be to grab the analytics information out of a read replica via exporting [3].

[1]  https://issuetracker.google.com/70756171

[2] https://issuetracker.google.com/120720131

[3] https://cloud.google.com/sql/docs/mysql/import-export/exporting

[Geert Van Landeghem]

Gordon, what's your use case?

I don't know what kind of permissions you need to configure triggers in postgreSQL, I saw messages in a forum that people were able to use that.

Your application could emit events when data is inserted, updated, deleted to pub/sub or kafka.

You could use kafka's JDBC source connector which polls the database but then you need to work with

• polling for changes in the database
• using timestamps (of insert, update & delete) & key fields
  
• logically delete records (flag record as deleted but don't physically delete the records)
  

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/e5d590b9-0257-4858-9105-498cf4e8312fn%40googlegroups.com.