Stuck connectors unable to send messages
Zach Walker
I have been working on this problem for a few weeks and have run out of ideas. See a description of our setup at the bottom. I don't know if this is necessarily a debezium specific problem, but that is the only producing connector we are running. I am hoping this community has the experience to help debug it.
Setting the log level of the WorkerSourceTask to TRACE does not give any additional log messages. This state can go on for several days without resolving itself. Restarting the connectors has been our only way to resolve the problem.
Notice:
1. There are 2 debezium tasks running on this worker
2. The number of messages outstanding for each connector is staying the same at each error message.
Our setup is as follows
Debezium 1.6 using the mysql plugin
- version - 6.1.1-ccs
- commit sha: c209f70c6c2e52ae
- connect workers - 96
- RAM - 32GB per worker
- Max Heap 28 GB per worker
- Attempting to run no more than 2 debezium tasks per worker
- connectors
- debezium - ~150
- Source Databases
- 200- 300 databases
- 150,000 tables across all databases
- 1,500,000 columns across all databases
- Source Databases
- other - snowflake (~30 tasks)
- debezium - ~150
Zach Walker
- According to the REST API, the debezium connectors and their tasks from the logs above are still in the RUNNING state
- No exceptions have been produced since the connectors were started
Gunnar Morling
Zach Walker
--
You received this message because you are subscribed to a topic in the Google Groups "debezium" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/debezium/B0-HIJk0VHo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/a2d11efa-64c4-4f58-a361-0181e4e476b7n%40googlegroups.com.
Zach Walker
I wanted to post an update on this and what we found to be the resolution of the issue in our case. Our problem and solution was different from any others we have seen people write up. The following solution and summary were put together by Brandon Stanley on our data engineering team here at Appfolio.
We were able to resolve the flush timeout issue by increasing the following Kafka Connect configuration: status.storage.partitions from 5 to 25 (https://docs.confluent.io/platform/current/installation/configuration/connect/index.html#connectconfigs_status.storage.partitions).
Here are the steps we took which prompted us to update the value:
We observed various metrics for our connectors and noticed that the buffer-available-bytes metric for all of our connectors that were stuck had reached 0 but never recovered. At first glance this was somewhat expected since we could see that data was backing up though it seemed like a symptom of the problem, and not the cause.
As per the documentation, buffer.memory is:
"The total bytes of memory the producer can use to buffer records waiting to be sent to the server. If records are sent faster than they can be delivered to the server the producer will block for max.block.ms after which it will throw an exception.
This setting should correspond roughly to the total memory the producer will use, but is not a hard bound since not all memory the producer uses is used for buffering. Some additional memory will be used for compression (if compression is enabled) as well as for maintaining in-flight requests."
We set the logging level to TRACE for the connectors for all producer loggers to see if we could find any exceptions that were being thrown related to the buffer memory.
This was done using the following command:
curl -s -X PUT -H "Content-Type:application/json" http://localhost:8083/admin/loggers/org.apache.kafka.clients.producer -d '{"level": "TRACE"}' | jq '.'
Then, we noticed the following message within the logs:
[2021-11-23 00:48:10,071] DEBUG [Producer clientId=producer-2] Exception occurred during message send: (org.apache.kafka.clients.producer.KafkaProducer) org.apache.kafka.clients.producer.BufferExhaustedException: Failed to allocate memory within the configured max blocking time 60000 ms.
Instead, we looked for other configurations that we had the ability to modify. We looked at the associated configurations for kafka connect and found that the number of partitions defaults to 5 based on this configuration.
Given the number of connectors/tables/topics for our use case, 5 partitions seemed too small. We decided to increase the number of partitions to 25 (based on the default configuration value for another internally managed topic: offset.storage.partitons). Once we increased the number of partitions to 25, both Failed to commit offsets and BufferExhaustedException messages were no longer thrown. In summary, our problem seems to have been that kafka connect needed further partitioning of the status storage topic to keep up when running as many workers and connectors as we are in the same worker group. After restarting all our connectors from scratch, we have not seen the same problem reoccur again.
Lingering questions and suggestions (For kafka connect, not specific to debezium)::
1. Why does the org.apache.kafka.connect.runtime.WorkerSourceTask execute loop produce no log output in this situation even while in TRACE? It's not clear to us what code path is being executed or where it is blocked.
2. The names of producer-1, producer-2, producer-3 would be more helpful if they included something about their function.
Hope this helps someone else.
--
You received this message because you are subscribed to a topic in the Google Groups "debezium" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/debezium/B0-HIJk0VHo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/a2d11efa-64c4-4f58-a361-0181e4e476b7n%40googlegroups.com.