Postgres publication tables list

[Shlomi Lanton]

Hey,

I'm using Postgres 10.13 on RDS, as recommended in the doc I'm trying to use the "pgoutput" plugin.

As mentioned in the section below, I understand that I need to manually create the publication with the superuser role. The thing that I'm not sure about is, once I create it can "debezium" modify it? if I set it up to "ALL TABLES" and then I want to change the connecter to some table, do I need to make the change with the superuser again? can I make the change only in the connector? should I change both?

The relevant text from the docs.

The name of the PostgreSQL publication created for streaming changes when using pgoutput.

This publication is created at start-up if it does not already exist and it includes all tables. Debezium then applies its own include/exclude list filtering, if configured, to limit the publication to change events for the specific tables of interest. The connector user must have superuser permissions to create this publication, so it is usually preferable to create the publication before starting the connector for the first time.

If the publication already exists, either for all tables or configured with a subset of tables, Debezium uses the publication as it is defined.

Thanks,

Shlomi

[Gunnar Morling]

Hi,

I'm not 100% sure I get what you're asking. If you have created the publication yourself upfront (as recommended), Debezium will not modify it in any way. If you e.g. set it up for ALL TABLES but then configure a connector which only has some tables in its include list filter, it will discard any change events for tables not matching that filter. On the Debezium side, support for filter config changes (still) is limited; while you can narrow it down -- excluding more tables than initially -- expanding will only work if you don't expect any snapshot to be done for those newly captured tables.

--Gunnar

[Shlomi Lanton]

Thanks for the replay.

Can you elaborate on "expanding will only work if you don't expect any snapshot to be done for those newly captured tables."?

In other words, what should be the process if I want to add a new table to Debezium?

For example:

In my DB I have the following tables: Table 1, Table 2, Table 3 and Table 4.

I create a publication in Postgres for only tables 1 and 2, and set those tables in the connector.

The connector is doing the initial snapshot and start streaming those tables.

After a few days I decide that I want to also track Table 4.

What steps should I take? can I add it to the existing Postgres publication? and how do I modify the connectors? should I create a new one?

Hopes that now my question is more clear,

Shlomi