Debezium for AWS RDS Postgresql version 9.4.15

[Pranav Agrawal]

My use-case is to pull data events from psql using debezium and then processing and storing it in hdfs. I am having issues installing debezium connector to connect to AWS RDS psql. I have kafka connect up and running with debezium connector installed and configured to connect to my psql instance. As per documentation in http://debezium.io/docs/connectors/postgresql/, there are certain config changes that needs to be made in psql rds instance for it to send events to debezium.

1.) set 'shared_preload_libraries' conf to plugin 'wal2json' and 'decoderbufs' -- AWS RDS Parameter Group doesn't allow setting these values.
2.) I need to create role with REPLICATION permission  -- AWS RDS doesn't allow to create this.
3.) will debezium work for postgres version 9.6+ only? -- I am using 9.4.15 version.

I am unable to configure because of these issues. Please help me with these issues to setup debezium for my use-case.

Thanks,
Pranav Agrawal

[Jiri Pechanec]

Hi,

RDS confiuration is described in http://debezium.io/docs/connectors/postgresql/#amazon-rds

wal2json plug-in is not supported in 9.4, see https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_PostgreSQL.html, it seems that wal2json is supported since 9.5

J.

[Ondra Žižka]

Ahoj Jirko,

I am struggling setting up wal2json in AWS. I have added the ParameterGroup, gave it to the RDS and restarted the RDS.

Now I want to add a user with REPLICATION LOGIN but can't - AWS's  PostgreSQL won't let me

CREATE USER replicant REPLICATION LOGIN CONNECTION LIMIT 1 ENCRYPTED PASSWORD 'replicant';

That needs SUPERUSER, which postgres user doesn't have and I can't get to rdsadmin user (that's Amazon's mgmt user).

Anyway. In the parameter group, wal_level and rds.wal_level are rejected.
So searched for it in the UI console and I can't see it.

When I run  SHOW ALL, wal_level is only hot_standby.

When I try ALTER SYSTEM SET wal_level = logical, I need SUPERUSER again.

So I'm kind of stuck. Debezium link you posted says

"""Verify that wal_level parameter is set to logical; this might not be the case in multi-zone replication setups."""

So I verified that it is not and I see no way to change it :)

Any hints please?

Ondra

[Jiri Pechanec]

Hi,

as is mentioned in the docs https://debezium.io/docs/connectors/postgresql/#amazon-rds

  - Use database master account for replication as RDS currently does not support setting of REPLICATION privilege for another account.

J.