Discontinuing wal2json support

[Gunnar Morling]

Hi all,

The Postgres community is going to discontinue support for version 9.6 in November this year [1]. Subsequently, any managed databases are likely to be upgraded soon, for instance, on AWS RDS any remaining 9.6 instances will be upgraded to newer versions in early 2022.

We are therefore considering to drop support for the wal2json logical decoding plug-in from the Debezium Postgres connector for release 1.8, due at the end of this year [2]. Supporting wal2json has added lots of complexity to the Debezium Postgres connector code base, and is an ongoing source of increased maintenance efforts.

We had added wal2json support to the Debezium Postgres connector initially as back then it was the only option for using Debezium with managed Postgres environments like RDS, where you cannot install Debezium's own decoding plug-in decoderbufs. With the advent of pgoutput in Postgres 10, the need for wal2json has largely diminished, and our recommendation has been for a while now to use pgoutput in managed environments, with decoderbufs being an alternative for on-prem.

In general, the different plug-ins are mostly equivalent in terms of functionality. One difference to note though is that pgoutput, unlike wal2json and decoderbufs, does not emit change events for tables without primary keys. As having such tables isn't a recommended practice, I think this should not be the primary driver for making that decision, though.

If you got any feedback or concerns about this planned removal of wal2json support, please let us know either by replying to this thread or by commenting on the Jira issue linked below.

Thanks,

--Gunnar

[1] https://www.postgresql.org/support/versioning/

[2] https://issues.redhat.com/browse/DBZ-3953

[Bernd Helmle]

Hi Gunnar,

Am Mittwoch, dem 01.09.2021 um 03:26 -0700 schrieb 'Gunnar Morling' via
debezium:

> In general, the different plug-ins are mostly equivalent in terms of
> functionality. One difference to note though is that pgoutput, unlike
> wal2json and decoderbufs, does not emit change events for tables
> without
> primary keys. As having such tables isn't a recommended practice, I
> think
> this should not be the primary driver for making that decision,
> though.

In the absence of a PK, this should be possible if you set the REPLICA
IDENTITY to either an explicit UNIQUE index or (if no such index is
available at all), FULL, which uses the whole row as an "identity".
Highly discouraged if someone doesn't really know the implications!

That said, i agree that this isn't a showstopper.

Thanks

Bernd

[Gunnar Morling]

Hi Bernd,

Thanks a lot for weighing in! Good point on using REPLICA IDENTITY with a unique index; I'll take a note in the DBZ-3953 issue to update the Debezium docs a bit in that regard.

--Gunnar

[Chris Cranford]

Hi M,

Yes, pgoutput can support large transactions in a similar way to that of wal2json's streaming modes.

Thanks,
CC

On 3/15/22 23:56, M Darling wrote:

Hi Gunnar,

I am confused whether the pgoutput plugin can support the scene where transactions are very large like wal2json_streaming or wal2json_rds_streaming? 

Thanks

M

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/61b7af2e-94f6-4b1d-9a21-8755e1fc0028n%40googlegroups.com.