How to change offset

[srikant dalai]

Hi All

I want to move the offset manually after java.lang.ArrayIndexOutOfBoundsException. My offset is at offset.storage.file.filename=/tmp/connect.offsets.. 

How to change the  the offset  here in the file based .

[jiri.p...@gmail.com]

Hi,

the offset file is serialized java.util,Map, see org.apache.kafka.connect.storage.FileOffsetBackingStore.save() So ideally you should write a simple Java app that would read the file, modify the content of the Map and than write it back.

J.

[Nathan Smit]

I had/have a similar requirement and so I created this Java thingy to update the offsets.dat file:

https://github.com/nathan-smit-1/HashmapEditor

If anyone wants to try it out or improve it please let me know.  It's very much hacked together, but I did some initial tests by setting to an earlier scn and it seems to work ok.

[jiri.p...@gmail.com]

Hi, this looks cool. Have you thought about making it a bit more generic - so ability to list the fields and values and ability to overwite arbitrary values?

If yes that would be very nice addition to debezium-examples repo.

J.

[Nathan Smit]

Hey Jiri, I will take a look and see.  I guess I could possibly have a command to write out the key and value and then have an option for the user to replace the value part of the pair with whatever they want?

[jiri.p...@gmail.com]

Hi,

yes, that's one option. printo all, print one, update one. Another approach would be printing the content as JSON and accepting JSON.

Both approaches are fine form my PoV.

J.

[David Brinton]

I also have need to update my offsets.dat after a power failure and a corrupted offsets file. I have noticed, though, that nath...@pepkorit.com's utility reports success but doesn't actually change the offsets file. The offset I want to change my file to is 14186561 so I'm passing `-scn 14186561`, but SCN isn't in my offsets file anywhere, instead mine has "pos". Is this something that has changed recently?

Here is my command and the output

C:\Users\me\source\HashmapEditor\target>java -jar HashmapEditor-1.0-SNAPSHOT.jar -offset_path "offsets.dat" -scn 14186561
The Object has been read from the file
Closing object
The Object has been written to the file
Closing object

I'm just wondering if I update the application to change the pos if that will be what I need? Not sure why my offsets file would be so different.

[jiri.p...@gmail.com]

IIUC you are using MySQL - each connector has its own offsets schema format.

J.

[Nathan Smit]

Hey David, as Jiri said above, each connector has a different format.  What I posted above was specific to the Oracle connector.  I am in the process of contributing an offset-editor which is a bit more user friendly and which has a frontend you can use to edit the offsets in a more granular way if you want to try it out:

https://github.com/nathan-smit-1/debezium-examples/tree/DBZ-6338/offset-editor

With the above it just extracts the whole offsets string and you can then edit any part of it you want and then save it again.

It's not part of debezium-examples as I still owe jiri some work on it that I haven't been able to get to yet :)