Can Debezium be used as an alternative to logical replication in Cloud SQL (PostgreSQL)?

[Daniel Franco]

It may be a newbie question, but I'm facing a very particular situation and I would be thankful if somebody could help me. It revolves around other technologies besides Debezium, so I wouldn't mind if I can't get the answer here.

I have to do a logical replication of my Cloud SQL instance to an external database and vice-versa. More specifically, the CloudSQL and the external instance will contain some master and slave tables that must be replicated between each database. However, PostgreSQL logical and external replication are currently not supported by the GCP. Then, I found out about Debezium.

Can I use Debezium to help me with this? Can I create one service to read the decoded CDC data from Debezium/Kafka and apply the changes to the external database and vice-versa?

Is it possible to capture the changes of some specific tables (not the whole DB)?

Since there are two databases in question here, would it be overkill to use Kafka + Debezium for this?

[Gunnar Morling]

Hi,

Am Dienstag, 19. Juni 2018 22:01:34 UTC+2 schrieb Daniel Franco:

It may be a newbie question, but I'm facing a very particular situation and I would be thankful if somebody could help me. It revolves around other technologies besides Debezium, so I wouldn't mind if I can't get the answer here.

I have to do a logical replication of my Cloud SQL instance to an external database and vice-versa. More specifically, the CloudSQL and the external instance will contain some master and slave tables that must be replicated between each database. However, PostgreSQL logical and external replication are currently not supported by the GCP. Then, I found out about Debezium.

Can I use Debezium to help me with this? Can I create one service to read the decoded CDC data from Debezium/Kafka and apply the changes to the external database and vice-versa?

Yes, that'd be a use case for using Kafka Connect and Debezium. You'd use Debezium on the source side and the Confluent JDBC sink connector on the sink side. Note though that Debezium requires a specific logical decoding plug-in (either ProtoBufs or wal2json) installed within the source Postgres, I'm not sure whether you have that flexibility with your cloud SQL provider (but e.g. on Amazon RDS, wal2json is installed by default). Would be cool if you could check this and report back what's available. We could add this to the docs.

Is it possible to capture the changes of some specific tables (not the whole DB)?

Yes, you can use whitelists/blacklists for specifically selecting the tables to be captured (see the connector docs for details). 

Since there are two databases in question here, would it be overkill to use Kafka + Debezium for this?

I think it's a good usage of these technologies. 

--Gunnar

[phil....@demandlogic.co]

Has there been any answer to this? We're looking to start using GCP, and in particular Postgres, but we're also very keen to make use of change data capture from Postgres, using something like Debezium. We would much rather use the Google SQL service to host the database, rather than deploying Postgres ourselves inside our own containers on GKE, but I've found no evidence of anyone using logical replication on a Google SQL Postgres instance.

Phil

[Jiri Pechanec]

Hi,

I went through Goggle's docs and it seems the (external) replication is still not supported which I believe is what incorporates logical decoding.

J.

[Shahnewaz Khan]

Cloud SQL feature request for this

https://issuetracker.google.com/issues/70756171