CDC with Postgres SQL to MySQL

[Avinash Omi Rai]

Hi All,

  I am new Siddhi/WSO2SI setup and I am trying to setup the application to get the data from Postgres to MYSQL. Below us the application I am working on:

--- Siddhi Application ----

@App:name('CDC')
@App:description('Capture Postgres Inserts and Updates using cdc')

@source(type = 'cdc', url = "jdbc:postgresql://localhost:5432/school", username = "*****", password = "****", table.name = "public.SweetProductionTable", operation = "insert,update", plugin.name = "pgoutput", snapshot.mode = "always",
    @map(type = 'keyvalue'))
define stream insertSweetProductionStream (name string, amount double);

@store(type = 'rdbms', jdbc.url = "jdbc:mysql://**.**.**.**:3306/<DB_NAME>?useSSL=false", username = "***", password = "****", jdbc.driver.name = "com.mysql.jdbc.Driver", table.name = "SweetProductionTable")
@primaryKey("name")
define table SweetProductionTable (name string, amount double);

@info(name = 'QueryonSweetProductionStream')
from insertSweetProductionStream
select *
update or
insert into SweetProductionTable for current events
    on SweetProductionTable.name == name;

--- Siddhi Application ---

When I am launching this application following out in the logs i can see

-- logs------

2022-08-18 07:47:59,732]  INFO {io.debezium.relational.RelationalSnapshotChangeEventSource} - Snapshot step 3 - Locking captured tables [public.sweetproductiontable]
[2022-08-18 07:47:59,732]  INFO {io.debezium.connector.postgresql.PostgresSnapshotChangeEventSource} - Waiting a maximum of '10' seconds for each table lock
[2022-08-18 07:47:59,750]  INFO {io.debezium.relational.RelationalSnapshotChangeEventSource} - Snapshot step 4 - Determining snapshot offset
[2022-08-18 07:47:59,750]  INFO {io.debezium.connector.postgresql.PostgresSnapshotChangeEventSource} - Creating initial offset context
[2022-08-18 07:47:59,750]  INFO {io.debezium.connector.postgresql.PostgresSnapshotChangeEventSource} - Read xlogStart at 'LSN{0/19705A0}' from transaction '4473'
[2022-08-18 07:47:59,751]  INFO {io.debezium.connector.postgresql.PostgresSnapshotChangeEventSource} - Read xlogStart at 'LSN{0/19705A0}' from transaction '4473'
[2022-08-18 07:47:59,751]  INFO {io.debezium.relational.RelationalSnapshotChangeEventSource} - Snapshot step 5 - Reading structure of captured tables
[2022-08-18 07:47:59,751]  INFO {io.debezium.connector.postgresql.PostgresSnapshotChangeEventSource} - Reading structure of schema 'school'
[2022-08-18 07:47:59,773]  INFO {io.debezium.relational.RelationalSnapshotChangeEventSource} - Snapshot step 6 - Persisting schema history
[2022-08-18 07:47:59,773]  INFO {io.debezium.relational.RelationalSnapshotChangeEventSource} - Snapshot step 7 - Snapshotting data
[2022-08-18 07:47:59,773]  INFO {io.debezium.relational.RelationalSnapshotChangeEventSource} -   Exporting data from table 'public.sweetproductiontable'
[2022-08-18 07:47:59,773]  INFO {io.debezium.relational.RelationalSnapshotChangeEventSource} -   For table 'public.sweetproductiontable' using select statement: 'SELECT * FROM "public"."sweetproductiontable"'
[2022-08-18 07:47:59,776]  INFO {io.debezium.relational.RelationalSnapshotChangeEventSource} -   Finished exporting 12 records for table 'public.sweetproductiontable'; total duration '00:00:00.003'

[2022-08-18 07:48:00,199]  INFO {io.siddhi.extension.io.cdc.source.listening.RdbmsChangeDataCapture} - Provided value for "op" : r is not supported.

--- logs----

 

But no entries are coming in MYSQL DB , only when the inserts are happening while the application is running is being pushed in MYSQL where as the vice versa is working (i.e. Date from MYSQL to Postgres)

Please guide me how can I approach the use case, any guidance is this regards is very valuable. And if this can be achieved or not.

[Avinash Omi Rai]

Hi All any pointers will be really helpful.

Regards,
Avinash

--
Siddhi homepage: http://siddhi.io/
Siddhi Github repo: https://github.com/siddhi-io/siddhi
 
You received this message because you are subscribed to the Google Groups "Siddhi-dev" group.
To post to this group, send email to siddh...@googlegroups.com
To unsubscribe from this group, send email to siddhi-dev+...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/siddhi-dev?hl=en?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "siddhi-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/siddhi-dev/0047IPJJUbY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to siddhi-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/siddhi-dev/126ea593-61eb-4533-b824-1506b50b602cn%40googlegroups.com.