Debezium & Postgres Master

[David Conde]

I have been trying out the Postgres connector, I am looking to do extraction from a large production database. I was reading that it is only possible to stream the changes from a master. What is the impact on the master of doing so and with is the recommended approach in this scenario? I do have a replica available to use, I would feel more comfortable using it but it sounds like it is not possible at the moment?

Thanks,

Dave

[Randall Hauch]

From the Debezium PostgreSQL documentation [1]:

Logical Decoding replication slots are only supported on master servers: this means that when there is a cluster of PostgreSQL servers, the connector can only run on the active master server. It cannot run on hot or warm standby replicas. 

I don't think there's any other reason, so if it is now possible to set up Logical Decoding replication slots on a follower server then you could certainly try to install the LD plugin and used the connector.

[1] http://debezium.io/docs/connectors/postgresql/

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+unsubscribe@googlegroups.com.
To post to this group, send email to debe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/f74b1eea-601a-471d-93e1-649baa8a307a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Horia Chiorean]

As Randall said, logical decoding was (at least until Postgres 9.6) possible only on master servers.

If I remember correctly, logical decoding was/is implemented as a separate process running within the main PostgreSQL server.

Some of drawbacks of logical decoding were:

• it can take up additional resources, since logical decoding plugins contribute to the overall server workload
  
• on Linux, if the logical decoding plugin crashed (either because of a bug in the code or some other reason), the entire server would reboot - not ideal in a production environment

Another limitation when running in a cluster was the fact that you could only have 1 plugin / logical decoding slot, so essentially you could not extend the active/passive topology to a plugin-level. In other words, should the server on which the plugin was running go down,

you'd have to explicitly restart another server with a logical decoding slot & plugin.

If newer versions of Postgres have done away with these limitations, I would definitively recommend using the plugin on a slave (ideally a hot standby) rather than on the master. Here one would need to make sure that the hot standby can have logical decoding configured so that once it's promoted to primary, it automatically continues to process the WAL from where it left off. Otherwise, you risk having a gap (from a plugin perspective) in the WAL data.

IMO it would be ideal if logical decoding would be supported in a master/slave fashion, where one could have one active plugin and multiple passive plugins.