Forward Compatibilty across rocksDB versions

[Radhika]

Our services use rocksDB v5.7.2, and we are looking to upgrade to rocksDB v8.3.2. However, our service is stateful, and receives continuous writes. We would like to accommodate for a rollback, but rollback fails with forward compatibility errors such as:

• Corruption: VersionEdit: unknown tag

• Invalid argument: Can't parse BlockBasedTableOptions:: checksum 

• Corruption: unknown checksum type 4

We were looking into making some code changes to allow for forward compatibility. This includes changing the format_version to 2, and also changing the v8 checksum to be kCRC32c by default. However, we would still face issues with reverting the patch to allow for forward compatibility. Is there a way we can guarantee forward compatibility? How do we upgrade rocksDB while allowing for rollbacks?

[Smt Asha Shrivastava]

This is very relevant concern and has waited for the response from Rocks-db developer community for months. Would appreciate a response on this.

Could someone please enlighten, how do clients manage the upgrade the Rocks-db library usage considering there have been 4-5 changes which are not backward compatible.

@Radhika, have you already sorted out your query? IMHO, there is a bigger problem when you upgrade to 9.0.0, I see the following in the release notes:

format_version=6 is the new default setting in BlockBasedTableOptions, for more robust data integrity checking. DBs and SST files written with this setting cannot be read by RocksDB versions before 8.6.0.

Thanks, Regards

Asha

[Adam Retter]

Could someone please enlighten, how do clients manage the upgrade the Rocks-db library usage considering there have been 4-5 changes which are not backward compatible.

I think it would help if you were a bit more specific with your question. What is the issue with upgrading that you are facing? Is it that the API has changed, or is it that you want to migrate to a newer supported data format?

Thanks, Adam.

--

Adam Retter

skype: adam.retter
tweet: adamretter
http://www.adamretter.org.uk

[Smt Asha Shrivastava]

Hi Adam

Appreciate your response.

I was curious to know, how do clients of RocksDb library (i.e. the users who are using RocksDb as their back-end store) manage the upgrade of the Rocks-db library versions (because they would have started their project with some base version of RocksDb library e.g. the original request on the thread from Radhika (  Our services use rocksDB v5.7.2, and we are looking to upgrade to rocksDB v8.3.2 ) ?

Considering there have been 4-5 changes (after 5.x.x) which are not backward compatible (ref: ), it made me skeptical to think how will the applications running with newer RocksDb versions read the SST files written using older formats. Is there a tool to help in migration of library OR to measure the impact of migration? Is there a tool for format conversion of SST files from one version to another?

Also, let's say a client application is using 7.9.0 and when the client upgrade to 9.0.0, I see there is an issue (i.e. as per the following from the release notes: format_version=6 is the new default setting in BlockBasedTableOptions, for more robust data integrity checking. DBs and SST files written with this setting cannot be read by RocksDB versions before 8.6.0.

Please correct me, if my understanding is not correct or I am missing something.

Thanks, Regards

Asha