Support for running against Cloud Pub/Sub emulator in Debezium Server

[Kristian Freed]

Hi,

I'm evaluating Debezium Server as a component for a new project and overall it fits the gap that I have currently quite nicely, for our use case subscripting to new events in Postgres and publishing to Cloud Pub/Sub using the outbox pattern.

I've gotten relatively far with this setup, but unfortunately have now realised that it appears impossible to use the local pub/sub emulator together with Debezium Server, and for this project I need to be able to replicate the production setup with a local Docker compose setup starting postgres, the pub/sub emulator, Debezium, etc.

This appears to be down to a quirk in how the Java SDK for pub/sub handles (or rather does not handle) the environment variables controlling use of the emulator. Normally, SDKs would simply detect the presence of an environment variable PUBSUB_EMULATOR_HOST and which to emulator settings (see https://cloud.google.com/pubsub/docs/emulator?hl=de#pubsub-emulator-java), but the Java SDK does not suport this and as a result, it fails to start as no real credentials are provided.

There's a ticket for this on the pub/sub SDK, but as this has been open since 2020, I'm not holding my breath there (https://github.com/googleapis/java-pubsub/issues/308).

Would it be possible to add support for this in the pubsub sink for Debezium Server? The simplest approach would be to explicitly check for PUBSUB_EMULATOR_HOST being present, and if so perform the extra few steps required as described in https://cloud.google.com/pubsub/docs/emulator?hl=de#pubsub-emulator-java to use the emulator.

Cheers,

Kristian

[jiri.p...@gmail.com]

Hi,

there is https://github.com/debezium/debezium/pull/3250

I hope it will be merged soon

J.