Migrating from Tokutek MariaDB/MySQL 5.5 release to MariaDB 10 / Percona 5.6

[Phil]

Hi guys,

Have been running with Tokutek's provided binaries for MariaDB 5.5 for a while now.

Looking to investigate an upgrade to MariaDB 10 (at this stage).  I understand there are differences in the TokuDB implementation in these non-Tokutek releases.

Is there a migration guide that covers the TokuDB implications?

Thanks.

[Phil]

I've given it a go (migrating from Tokutek's mariadb 5.5.41 release to latest MariaDB (10.0.19), and one thing I'm getting for tokudb tables is these warnings:

150515 15:23:00 [Warning] ./mydatabase/mytable.frm: invalid value 7 for the field row_format

A number of these warnings appeared with the same format.. the invalid value ranges from 7 to 10, depending on the table.

Things seem to work, but I worry the compression stuff may not be working properly.

Any ideas?

Thanks,

Phil

[Rik Prohaska]

Hello Phil,

Tokutek's MariaDB 5.5 is incompatible with MariaDB's 5.5 and MariaDB's 10.0 in the following ways.

1.  The user interface for clustering keys is different.  Tokutek added clustering to the SQL parser.  MariaDB uses the index options extension.  As a result, the FRM files are incompatible WRT how clustering keys are stored.

2. The user interface for compression is different.  Tokutek added its compression options to the row_format variable,   MariaDB uses the table options extension.  As a result, the FRM files are incompatible WRT how the table compression is stored.

3. Every plugin gets a plugin number when it registers with MySQL during plugin initialization.  This plugin number is not the same on the various TokuDB releases for historical reasons.  A long time ago, TokuDB was assigned a dynamic plugin number.  This number could change from one instance to another depending on whether or not some other dynamic plugin was loaded prior to TokuDB.  Now, TokuDB uses a fixed plugin number.  But that number is different for Tokutek's MySQL, and Tokutek's MariaDB, and probably MariaDB's distros.

The plugin number is stored in the FRM files in one place for regular tables and in another place for partitioned tables.  MySQL has bugs WRT drop table if the stored plugin number is different than the plugin number actually in use. 

4. Tokutek's MariaDB 5.5 has a different feature set compared to MariaDB's 5.5 and 10.0 releases.  For example, there is no read free replication on MariaDB's 5.5 or 10.0.

So, there can be issues migrating between the various TokuDB distro's.

There are no migration tools now.  Unfortunately, dump and reload is the only safe way to do the migration.

[Rik Prohaska]

The primary problem is the FRM files.  If one could generate new FRM files for the tokudb tables, and delete the copy of the FRM data stored in tokudb's status dictionaries, then the rest of the tokudb data files are binary compatible.  A tool to delete rows out of the status dictionaries could be written fairly quickly if you want to play with it.

[Phil]

Thanks for the detailed answer Rik!

I assume the situation is very similar for Percona 5.6 (except that it does support RFR?)

I found this page:
https://mariadb.com/kb/en/mariadb/tokudb-differences/
Does that look right with respect to features? (apart from not mentioning RFR).

Also, I notice on this MariaDB page about TokuDB:
https://mariadb.com/kb/en/mariadb/tokudb/
It suggests only Ubuntu 12.10+ is supported.. any ideas why 12.04 wouldn't be supported?  MariaDB certainly offers version 10 on that platform.. hmm.

Thanks.

Phil

[Rik Prohaska]

On Saturday, May 16, 2015 at 3:03:07 AM UTC-4, Phil wrote:

Thanks for the detailed answer Rik!

I assume the situation is very similar for Percona 5.6 (except that it does support RFR?)

Yes.  I have a prototype migration tool that strips the FRM from all of the tokudb status files.  The migration process would be: 1) dump the schemas 2) translate the table defs to MariaDB format 3) strip the FRM from the tokudb status files 4) bring up mariadb 10 and create the tables with the new schema defs 5) shutdown mariadb 6) replace the tokudb files with the stripped tokudb files 7) bring up mariadb with the stripped tokudb files

I found this page:
https://mariadb.com/kb/en/mariadb/tokudb-differences/
Does that look right with respect to features? (apart from not mentioning RFR).

I have a RFR patch for mariadb 10 that could be used.

 

 

Also, I notice on this MariaDB page about TokuDB:
https://mariadb.com/kb/en/mariadb/tokudb/
It suggests only Ubuntu 12.10+ is supported.. any ideas why 12.04 wouldn't be supported?  MariaDB certainly offers version 10 on that platform.. hmm.

Need to ask Mariadb folks.

[Phil]

On Sunday, May 17, 2015 at 1:09:35 AM UTC+10, Rik Prohaska wrote:

Yes.  I have a prototype migration tool that strips the FRM from all of the tokudb status files.  The migration process would be: 1) dump the schemas 2) translate the table defs to MariaDB format 3) strip the FRM from the tokudb status files 4) bring up mariadb 10 and create the tables with the new schema defs 5) shutdown mariadb 6) replace the tokudb files with the stripped tokudb files 7) bring up mariadb with the stripped tokudb files

 

I tried doing an alter table <blah> engine=tokudb compression=tokudb_fast; to see if it rewrites the data in the accepted format.

The warning no longer appears when it loads the table.  Could this be an alternative?  Or is the plugin number thing still an issue here.

Phil

[Rik Prohaska]

'alter table engine=tokudb' may be a full table rebuild.  If so, then the FRM will be regenerated.

Unfortunately, the FRM files generated by Tokutek's MySQL and MariaDB releases are not compatible with the FRM files generated by MariaDB 10.  Here is an example.  Suppose that I have a simple tokudb table with a clustering key created with tokutek's mariadb 5.5.41.  If I copy the tokudb data files to a mariadb 10.0.19 installation, create table says that the index is FULLTEXT, and will not alter it.

MariaDB [test]> show create table test.s;

| s     | CREATE TABLE `s` (

  `x` int(11) NOT NULL DEFAULT '0',

  `y` int(11) DEFAULT NULL,

  `z` int(11) DEFAULT NULL,

  PRIMARY KEY (`x`),

  KEY `y` (`y`),

  FULLTEXT KEY `z` (`z`)

) ENGINE=TokuDB DEFAULT CHARSET=latin1 `compression`='tokudb_zlib' ;

MariaDB [test]> alter table s engine=tokudb;

ERROR 1464 (HY000): The storage engine TokuDB doesn't support SPATIAL indexes