Debezium Connector stuck during snapshot create

[Carole Godfrey]

Connector does not successfully create a snapshot
Seeing the following WARN message in Debezium logs

[Worker-066482f50e37b3ec3] [2024-02-29 10:20:11,647] WARN [Debezium-library-connector|task-0|offsets] Couldn't commit processed log positions with the source database due to a concurrent connector shutdown or restart (io.debezium.connector.common.BaseSourceTask:362)

We are running version of 2.3.4 of Debezium and MSK Connect.

The connector did not restart

Last Offset topic was written at 2/28/2024, 15:59:49 
{
"last_snapshot_record": false,
"lsn": 16248318296,
"txId": 422522693,
"ts_usec": 1709153928815673,
"snapshot": true
}

Any suggestions on how to proceed or further troubleshoot?

Was looking into a similar post
https://groups.google.com/g/debezium/c/EJwZVkovtgg

Thank You In Advance!

[Chris Cranford]

Hi Carole,

Can you share the full connector logs?

Thanks,
Chris

--
You received this message because you are subscribed to the Google Groups "debezium" group.
To unsubscribe from this group and stop receiving emails from it, send an email to debezium+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/c281296c-8707-4ab6-a711-40117b17f83fn%40googlegroups.com.

[Carole Godfrey]

HI Chris,

Attached are the connector logs

Thank You,
Carole

[Carole Godfrey]

And to note -- 1 correction

Last offset topic was written at 

2/29/2024, 07:44:29

{

"last_snapshot_record": false,

"lsn": 16248318296,

"txId": 422522693,

"ts_usec": 1709153928815673,

"snapshot": true

}
Thank You

[Chris Cranford]

Hi Carole -

I'd suggest keeping an eye on this conversion [1] on Zulip.

Our preliminary analysis indicates that there could be a race condition where BaseSourceTask's updateLastOffset is called for a specific record's offset and the Kafka Connect runtime concurrently performs a commit flush. Generally speaking, I would say there really isn't a cause for concern here as the commit flush will simply be delayed until the next available retry or when the task stops, whichever comes first. We are looking into ways that we can better handle this so that the warning message doesn't raise user concerns.

Hope that helps,
Chris

[1]: https://debezium.zulipchat.com/#narrow/stream/348104-community-mysql/topic/Couldn't.20commit.20processed.20log.20positions/near/424872700

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/d16bfe0f-ee39-42ee-960e-4be9261deb89n%40googlegroups.com.

[Chris Cranford]

Hi Carole -

I should also point out that it would seem like you had a re-balance happen in the middle of your massive snapshot. You should try and refrain from having any type of re-balance occur that would cause the connector task to restart during the snapshot of your database's magnitude.  If that's not possible for you, then you may need to reconsider using incremental snapshots and not taking the initial snapshot.

Chris

[Carole Godfrey]

Thank you Chris for all of these details.

As far as Im aware, a re-balance did not happen
Are there any best practice to prevent a re-balance or connector restart and avoid this situation?

I have been unsuccessful trying to create an initial snapshot -- As I understand with a connector restart It will restart the initial snapshot once again until successful
Is there a way to accomplish the equivalent of an inital snapshot with incremental snapshots

[Chris Cranford]

Hi Carole -

A connector deployment that relies on incremental snapshots rather than the initial snapshot will be deployed setting "snapshot.mode" as "schema_only". In this case only the schema structure information is read at deployment time and the connector immediately transitions into the streaming phase to read changes from the database transaction logs.  In addition, the connector should be configured with the necessary incremental snapshot configuration [1] which can vary slightly depending by connector. Once the connector has started, you send a signal that will be consumed by the connector's runtime and it will begin capturing the historical data from the specific tables incrementally and concurrently while streaming changes from the transaction logs. This process is resumable so if a re-balance happens at any point, the connector will resume where it left off.

You can see the signaling setup documentation [2] for more details.

Thanks,
Chris

[1]: https://debezium.io/documentation/reference/2.6/connectors/mysql.html#debezium-mysql-incremental-snapshots
[2]: https://debezium.io/documentation/reference/2.6/configuration/signalling.html#debezium-signaling-ad-hoc-snapshots

To view this discussion on the web visit https://groups.google.com/d/msgid/debezium/7c760445-8b58-4fd9-9cd1-8b6636f0990fn%40googlegroups.com.

[Carole Godfrey]

Thank you again for these details Chris!