Heartbeat table for postgres connector

[Jagat Patel]

Hello,

I am using these heartbeat configuration on the postgres connector as documented in https://debezium.io/documentation/reference/connectors/postgresql.html#postgresql-wal-disk-space

"heartbeat.interval.ms": "...",

"heartbeat.topics.prefix": "...",

"heartbeat.action.query": "....",

I have created PUBLICATION on the postgres database that is being monitoring and I have only added few tables to this PUBLICATION since I do NOT want to monitor all tables.

Do I need to add the heartbeat table to the PUBLICATION as well ? 

If not then how does debezium get the change event from the postgres database that it publishes to the heartbeat topic ?

Thanks,

Jagat

[jiri.p...@gmail.com]

Hi,

yes you must add it to the publication too.

J.

[Jagat Patel]

Hi J.

Thank you for your reply! I will add the heartbeat table to publication. One thing I am noticing is even if I do *not* add heartbeat table to the publication, I still see heartbeat change events in the kafka topic. I double checked that the publication do not have heartbeat table by executing this query: "select * from pg_publication_tables". Also I confirmed that my connector is using "pgoutput" plugin with my publication.  Is this behaviour expected ? 

Jagat

[Gunnar Morling]

Hi,

Yes, this is expected, as the term/concept "heartbeat" is a bit overloaded. The core heartbeat feature [1] regularly emits messages to the heartbeat topic, allowing to acknowledge processed WAL offsets also in case only events in filtered tables occur (this is what you observe). Heartbeat action queries [2] (which require the table and inclusion in publication) are useful to address situations with multiple databases, where the connector is receiving changes from one database with otherwise no/low traffic, again allowing to acknowledge offsets in this case.

--Gunnar

[1] https://debezium.io/documentation/reference/connectors/postgresql.html#postgresql-property-heartbeat-interval-ms

[2] https://debezium.io/documentation/reference/connectors/postgresql.html#postgresql-property-heartbeat-action-query 

[Jagat Patel]

Hello Gunnar,

ok, that explains the behaviour I am seeing. I was under impression that the change in the heartbeat table will send the WAL updates which will send heartbeat message to the kafka topic.

Thanks!

Jagat

[Don Seiler]

Good afternoon,

Would it be reasonable to update the documentation to explicitly state that the heartbeats table needs to be part of the publication to be of any real use? Perhaps it's obvious to some that are more fluent in Debezium and/or logical replication but we struggled for quite a few days over this one.

Going further, would it be entirely unreasonable to suggest that the heartbeats table be forced to be in the publication? Is there a strong case for excluding it? I can't imagine there is a lot of overhead even on a busy system if it's only updating every 10 seconds or so. Perhaps there are many cases I haven't thought of.

[Don Seiler]

I've created a PR to add a line about this [1]. Seems like there are a few hoops to jump through to actually contribute. I'll see if I can come back to this later and set up a Jira account and create an issue for it, and then update my commit messages.

[1] https://github.com/debezium/debezium/pull/4946

[Chris Cranford]

Thanks Don -

I think with regard to your latter point, there isn't currently a separate configuration that explicitly says "this table is the heartbeat table", so I don't believe forcing that from the connector's point of view would be possible.  We can do somethings like this with the signaling subsystem because that table is supplied in a separate configuration unlike heartbeats.

As for the PR, you can also update the commit with the "[docs]" prefix if its mostly a documentation update to avoid needing to write a Jira if that's easier.

Thanks,
Chris

NOTICE: This electronic mail message and any files transmitted with it are intended
exclusively for the individual or entity to which it is addressed. The message,
together with any attachment, may contain confidential and/or privileged information.
Any unauthorized review, use, printing, saving, copying, disclosure or distribution
is strictly prohibited. If you have received this message in error, please
immediately advise the sender by reply email and delete all copies.

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/2d731bb0-212b-4f01-a88d-754f02310076n%40googlegroups.com.

[Don Seiler]

Chris,

Thanks for responding. You are right that it probably makes sense to leave the second point alone. I've come to learn a little more from our app team about how Debezium was set up initially.

I will update my PR commits as you've suggested. Thanks again!

Don.

[Don Seiler]

I've squashed my commits into 1 with the [docs] prefix now.

Thanks!