XStream outbound sever in a RAC environments

[Kiril Piskunof]

Hi,

I’m working on setting up Debezium with Oracle XStream in an Oracle RAC environment, and I have a question about how this is supposed to work when use_rac_services is set to Y.

When use_rac_services=Y, the outbound server is pinned to a specific RAC node. Debezium must connect directly to that node for XStream to work. If the connection is made to the wrong node using SCAN, we get the following error:

oracle.streams.StreamsException: ORA-26913: must connect to instance 3 where XStream Outbound server "..." is running

We can configure the connector to connect to a specific node, which works in the steady state. However, I’m unclear how this is intended to work in a failover scenario. If the queue table moves to a different node, the outbound server will also move.

How is Debezium expected to handle this situation? Is there a recommended approach for reconnecting to the correct node automatically when the outbound server moves, or is manual reconfiguration unavoidable?

[Chris Cranford]

Hi -

That's a great question and it touches on specific subtle aspects of Oracle RAC with Oracle XStream.

    1. Make sure the outbound server is associated with a dedicated service, i.e. XSTREAM_SERVICE.
    2. On the client side with Debezium, utilize `database.url` in the connector configuration instead of `database.hostname`.

For (2), the `database.url` is a TNS connection string that defines the following:

    1. Enable fail over by setting (FAILOVER=ON)
    2. Disable load balancing so that Debezium connects to the explicit host that provides the service (LOAD_BALANCE=OFF)
    3. Specify the address list with all the SCAN hosts and port mappings
    4. Specify the connect data to use the specified service name from above, i.e. XSTREAM_SERVICE

With this set up, Debezium will only ever connect to the host that provides the XSTREAM_SERVICE at that time. If at any point the RAC node is taken down or if the service migrates to another node, Debezium will stop and restart, establishing a connection with the node that is now responsible for offering that service.

In addition, if you want to support Fast Application Notification (FAN) in Debezium, this can be done in the connector configuration:

    driver.oracle.ons.nodes=host1:port,host2:port
    driver.oracle.net.FAN_ENABLED=true

Hope that helps.
-cc

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/debezium/1850f1cb-b8a2-4cf9-a8f3-8fc1d6a14467n%40googlegroups.com.

[Kiril Piskunof]

Thanks Chris, this is very helpful.

My setup is a bit complicated. We rely heavily on SMTs and use a single outbound server per table, so there are quite a few of them. Creating a few services for groups of outbound servers is possible, but it would require propagating those groupings into the deployment configuration. Currently, all connectors share the same connection configuration.

I’m wondering if a more dynamic approach is viable. The idea would be to keep all connectors using the same connection details, but before starting a connector start up script would run a query to find the network name for the outbound server and update the connector config to connect to that node:

SELECT network_name FROM all_queues q JOIN all_xstream_outbound ob ON q.owner = ob.queue_owner AND q.name = ob.queue_name WHERE server_name = '<server_name>';

Is this a reasonable approach?

[Chris Cranford]

Hi -

Sorta, but this would only work for the initial deployment and not if the task fails & restarts due to service migration.

I have a slightly different suggestion, and that would be to bake this select into a custom ConfigProvider. Have the provider perform the resolution and then construct the `database.url` value with the right service name. Then you can continue to maintain the same connector configuration but using secrets so that the `database.url` value is resolved dynamically. This would work for both the initial deployment and for task restarts.

Jiri did an excellent write up about using secrets and ConfigProvider(s) in Debezium's configuration on the blog [1] that you might use as a reference for what I am suggesting. The only difference is the blog uses the standard FileConfigProvider, but that can be swapped out for a custom implementation as I suggested quite easily.

Let us know if you have any questions.
-cc

[1]: https://debezium.io/blog/2019/12/13/externalized-secrets/

To view this discussion visit https://groups.google.com/d/msgid/debezium/c355efc5-e9b5-4564-8838-bd6dc58ff2e3n%40googlegroups.com.

[Kiril Piskunof]

Thank you Chris, we will give this a go.