Migration of System to Vitess

[Harshit Gangal]

Hi,

I want to understand, if there is a said process to migrate the existing single database to vitess based cluster.

I did not find any documentation around it.

If there is any tool or script present which will do it.

Thanks,

Harshit

[Anthony Yeh]

We don't have a walkthrough yet for migrating an existing database into Vitess. We're working on a migration guide based on a sample app.

The general procedure will be to get to a state where the database is accessible both directly and through Vitess, then switch app servers to connect through Vitess.

For example, the following list outlines the steps for migrating an unsharded database. Once you've migrated onto Vitess, the steps for going from unsharded to sharded would be the same as our existing Resharding Guide.

If you have more specific questions, we'll probably need to know more about your setup to give recommendations.

1) Add Vitess users and grants on your existing database.

The default values can be found here:

https://github.com/youtube/vitess/blob/master/config/init_db.sql#L18

2) Start up zk/etcd and vtctld.

The same as for a from-scratch Vitess cluster.

3) Start up vttablet alongside each mysqld.

Instead of starting a fresh vttablet+mysqld combination, just start a vttablet on the same machine as each of your existing mysqld servers.

You'll need to add some extra parameters to tell vttablet the database name to assume. On a from-scratch Vitess keyspace, we call the MySQL database "vt_"+keyspaceName by default, and you probably don't already have your database named like that. :)

You'll also need to tell Vitess which tablet is the master, which you can do with the TabletExternallyReparented command. This is different from InitShardMaster (used in from-scratch setup) in that TabletExternallyReparented does not touch whatever MySQL replication you have set up - instead, it informs Vitess that you have already set up replication yourself.

4) Start up vtgate(s).

Same as from-scratch setup.

5) Plug in Vitess client.

Develop a plugin for the data access layer of your app that uses the Vitess client to connect to vtgate. We are working on native drivers for things like JDBC and PDO that would make this a more seamless switch.

6) Switch app servers over to Vitess client.

Once you've got the app working on the Vitess client, you can switch over your app servers to Vitess mode one by one.

--
You received this message because you are subscribed to the Google Groups "vitess" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vitess+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexander Medvednikov]

Hi Anthony,

Thanks for this guide.

Is it possible to run vttablet and the corresponding mysqld on separate machines?

[Anthony Yeh]

The vttablet process is designed as a sort of add-on to mysqld, so it needs to be on the same machine. For example, it needs access to the MySQL data directory for backup/restore, and it talks to mysqld over a local UNIX socket to minimize communication overhead and latency.

[Alexander Medvednikov]

That makes sense. Thanks for explaining.

[Alexander Medvednikov]

Hi,

Could you please give an example of how to actually start vttablet alongside mysqld?

We've been struggling with this.

We were using vttablet-up.sh as a reference.

As I understand, mysqlctl starts and initializes mysqld, so we don't need to run it, since we already have MySQL running with all our existing databases and tables.

It also creates /vtdataroot/vt_0000000100/my.cnf. Do we need to modify this file? We tried to run mysqld with this configuration file without any luck, and that's why running vttablet also failed.

Is there any docs on how to run vttablet? Or should we use vtctl InitTablet?

Sorry for so many questions, but there doesn't seem to be any examples of doing this. This is really crucial for us, as we are planning to use Vitess with our huge MySQL infrastructure.

Thanks a lot.

On Thursday, November 19, 2015 at 7:36:25 AM UTC+1, Anthony Yeh wrote:

[Anthony Yeh]

It's hard to give a concrete example because the idea is to start with your existing mysqld config and tweak it so it has the features that vttablet expects.

You're right that you should skip mysqlctl when attaching vttablet to an already-running mysqld. Since you're not using the my.cnf generated by mysqlctl, you will probably need to tell vttablet where to find the my.cnf for your already-running mysqld, which you can do with the -mycnf-file flag. Then vttablet will try to read the my.cnf file to figure out where the mysqld socket file and data directories are.

The examples/local/vttablet-up.sh script is the closest we have to documentation on vttablet flags, since it shows the minimal set needed to get started. The InitTablet command is not meant for actually running the vttablet process; it only refers to initializing the tablet record in the topology service. InitTablet is optional now because we prefer to instead pass the -init_keyspace and -init_shard flags to vttablet, which will let the tablet initialize itself in topology at startup.

If you'd be willing to share your existing mysqld config and vttablet logs, I can try to reproduce the issues you're having. Feel free to reply directly (eni...@google.com) if you don't want it on the public list. We can also set up a video call to discuss more specifics of your migration scenario.

Anthony