[Cluster Links
Jamar Lizarraga
Each node in a Redis Cluster maintains a pair of long-lived TCP link with each peer in the cluster: One for sending outbound messages towards the peer and one for receiving inbound messages from the peer.
Confluent Enterprise now includes a feature called Cluster Linking that syncs dataand metadata from Confluent Platform, Confluent Cloud, or Kafka clusters to Confluent Platform and Confluent Cloud clusters.
Cluster links and mirror topics can be created and managed using either a REST APIin the Kafka REST v3 Proxy, or using CLI commands in a terminal.This section describes the commands for creating and managing cluster links and mirror topics.
Configuration options for the cluster links are available as valuesfor flags on the kafka-cluster-links command. Some of these optionsare shown below in the context of command examples. A full list is providedin reference form in Link Properties.
(Required) The connection string for the broker(s) in a cluster in the form host:port (which can be a comma-separated list for multiple brokers).You must specify the destination cluster where you plan to create mirror topics.The destination cluster must be running Confluent Platform 7.0.0 or later, which is required to support Cluster Linking.
The only exception for a source-initiated link (recommended for Cluster Linking from a Confluent Platform source to a Confluent Cloud destination).In that case, you would create a cluster link object twice and the second time requires executing kafka-cluster-links with the source cluster as the bootstrap server.
(Required) One of the following parameters must be provided (not both) to specify how the destination clustershould communicate with the source. The available configurations are those that would be used to configure a client,including the required bootstrap.servers and other necessary security and authorization properties.
You can list existing cluster links. The command returns the link name, link ID(an internally allocated unique ID), the cluster ID of the linked cluster, and whetherthe linked cluster is available or not.
To confirm that your new configurations have been applied, use cluster.link.paused to pause and unpause the link to restart it.To learn more, see cluster.link.paused in Configure Cluster Linking on Confluent Platform.
A comma-separated list of configs to override when creating the mirror topic. Eachconfig to override should be specified as name=value.For more information about which configurations can be set on a mirror topic,see Configurations in Mirror Topics.
To list mirror topics on a cluster across all cluster links, use kafka-mirrors --list. This commandcan either list the mirror topics for a specific cluster link, or can list all mirror topics across allcluster links on the cluster.
(Required) The connection string for the broker(s) in a cluster is in the form host:port (which can be a comma-separated list for multiple brokers).You must specify the destination cluster where you plan to create mirror topics.The destination cluster must be running Confluent Platform 6.0.0 or later, which is required to support Cluster Linking.
You can use either the kafka-mirrors --failover or kafka-mirrors --promotecommand to stop mirroring on a topic. Both commands permanently convert the mirror topicinto a regular, writable topic, but each is designed for a specific use case.
You can perform promote and failover commands on multiple topics at the sametime by passing in a list of topic names using the --topics flag, and a comma-separatedlist of topic names. Even if you are only promoting 1 topic, you still use the --topics(plural) flag. For example:
Use kafka-mirrors --promote to stop mirroring and convert a mirror topic to aregular topic in a graceful process typically appropriate for migration scenarios.This command checks that there is 0 lag between the source topic andthe mirror topic, and does one final sync of the metadata (consumer group offsetsand topic configs) before converting the mirror topic into a regular topic.
Here is an example of using kafka-mirrors --failover to stop mirroring ona topic called example-topic which uses a link named example-link.The link will be inferred; it should not be included in the command, as shown below.
To migrate a consumer group across the link, set consumer.offset.sync.enable=truein your link configuration, specify a group filter in a JSON file, and pass the name of thefile as the value for the --consumer-group-filters-json-file flag on the CLI commands.You can set this at the time you create the link, or as an update to an existing configuration.
Cluster Linking enables you to directly connect clusters and mirrortopics from one cluster to another.Cluster Linking makes it easy to build multi-datacenter, multi-region,and hybrid cloud deployments. It is secure, performant, tolerant ofnetwork latency, and built into Confluent Server and Confluent Cloud.
Cluster Linking is included as a part of Confluent Server. There are no additional or other licensing costs for Cluster Linkingon Confluent Platform outside of the cost of the Confluent Enterprise Enterprise (Subscription) license. Following are the requirements for and supported featuresof Cluster Linking.
Use Case: Easily create a persistent and seamless bridge from on-premise environmentsto cloud environments. A cluster link between a Confluent Platform cluster in your datacenter and a Confluent Cloudcluster in a public cloud acts as a single secure, scalable hybrid data bridge that can be usedby hundreds of topics, applications, and data systems. Cluster Linking can tolerate the high latencyand unpredictable networking availability that you might have between on-premise infrastructureand the cloud, and recovers from reconnections automatically. Cluster Linking can replicate databidirectionally between your datacenter and the cloud without any firewall holes or special IP filtersbecause your datacenter always makes an outbound connection. Cluster Linking creates a byte-for-byte,globally consistent copy of your data that preserves offsets, making it easy to migrate on-premise applicationsto the cloud. Cluster Linking built into Confluent Platform and does not require any additional components to manage.
Use Case: Stream data between the continents and regions where your business operates. Unifydata from every region to create a global real-time event mesh. Aggregate data from differentregions to drive the real-time applications and analytics that power your business. By making geo-local readsof real-time data possible, this can act like a content delivery network (CDN) for your Kafka eventsthroughout the public cloud, private cloud, and at the edge.
Customer Success Story (video): Real-Time Inter-Agency Data Sharing With Kafka,Kafka and Cluster Linking have transformed how government agencies share data: in real-time with faster onboarding of new data sets,real-time event notification, reduced cost for data sharing, and enhanced and enriched data sets for improved data quality.
Because Cluster Linking fetches data from source topics, the first scalingunit to inspect is the number of partitions in the source topics. Having enoughpartitions lets Cluster Linking mirror data in parallel. Having too fewpartitions can make Cluster Linking bottleneck on partitions that are more heavily used.
Confluent proactively monitors all cluster links in Confluent Cloud and willperform tuning when necessary. If you find that your cluster link is not hittingthese limits even after a full day of sustained traffic, contact Confluent Support.
Node 02 went fine, even saw the check mark in the cluster update overview, the 2nd node then began to be stuck. and i recently saw a line which said firmware update of the nic e0a failed ? Could that be the reason why the cluster interconnect
Actually we just wiped it again and started all over. The good thing is, its all working now though. Didnt have time to play around longer since i need it. Links are up, 9.5 P1 (its a Lenovo AFF220 Think System) cause i cant get P2 yet. But its all back to normal, im gonna try to Cluster Update it again when i get P2 and will report if i see something special or if the failover again fails.
Just had this issue after upgrading to 9.5P3 from 9.4P3. After 2 days of trying to figure it out with Netapp Engineers. The solution was simply to issue the power cycle command (not a node reboot) in the BMC... I like simple solutions but COME ON two days and it was that simple, I feel stupid... Glad it was that easy though...
The VCS are being run on a Solaris machine v440. The VCS version is 4.0 on Solaris 9, I know it's old & EOL. Im just hoping to find and pinpoint the soloution to this problem.
The VCS heartbeat links are running on 2 seperate Vlans. This is a 2 node cluster.
Recently the old switch was taken out and a new switch CISCO 3750 was added. The switch shows the cables are connected and I am able to see link up from the switch side.
The links in ce4 of both servers are not linking. Any ideas besides faulty VLAN? How do I test the communications on that particular VLAN?
Here are the results of various commands, any help is apperciated!
Thank you!
#lltstat -n
LLT node information:
Node State Links
0 node1 OPEN 1
* 1 node2 OPEN 2
I checked on the switches and the connections are on the same VLAN and it is active.
ce6 heartbeats are in VLAN 1 and ce4 heartbeats are in VLAN 6. How can I tell which on is the primary heartbeat and which is the secondary heart beat. I'm clueless as to what to do next.
If ping is not working, then either there is an issue with the connection between the interfaces, or the switches may be blocking ping, so assuming that both ce4 and ce6 both have the same firewall configurations, unplumb the IPs from ce4 and plumb them onto ce6 - if ping works using ce6 then this proves the issue is with the network (this proof is usually needed if you have a separate network team as I have alway found the network team say the issue is the O/S until you can prove otherwise).
795a8134c1