How to move all keys from a Redis database to another

[Ronny López]

Hello,

We need to move all the keys in a Redis database to another. For
example, from DB 0 to DB 1, in the same redis instance.

We are upgrading the application, and we need to perform certain
operations with the data, but keep the application running. For this
reason, we are going to perform the changes over a new database,
cloned from the current one, and then change the application to use
this new database.

Another use case is unit testing. We need to clone the database to run
the tests with the application data, but in a test database.

So, how this can be done? What are the best practices?

Thanks.

[Salvatore Sanfilippo]

Hi Ronny,

before to continue actually replying your question, there are much
better ways to handle your use case, especially the following:

1) Start a new instance of Redis.
2) Make the new instance a clone of your production instance using:
SLAVEOF <host> <port>
3) When the replication finished cloning the instance (you can see
that from the logs of the slave), turn the new instance into a slave
with: SLAVEOF NO ONE.
4) At this point manipulate the slave instance as you wish.
5) When you are ready finally either switch the client configuration
to use the new instance, or, use replication again (SLAVEOF and so
forth) in order to clone the new data into the old instance. But this
will cause some downtime, so switching the instance is the way to go
IMHO.

make sure to do a backup of your most recent RDB file before doing all
that. Even if you are using AOF persistency you can easily create an
RDB backup calling the BGSAVE command.

Note that while you alter the data on the new instance all the changes
arriving on the old instance are not replicated since we disconnected
the two instances with SLAVEOF NO ONE. But this also happened in your
scenario with two databases.

Now if you still want to clone the database 0 into 1 please reply here
and I'll tell you the (not trivial) way to do this efficiently, that
basically involves rewriting an AOF log, concatenating two copies of
it but with the second copy having the SELECT command altered to
select DB 1.

Cheers,
Salvatore

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

--
Salvatore 'antirez' Sanfilippo
open source developer - VMware

http://invece.org
"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele

[catwell]

You might also find this thread useful:
http://groups.google.com/group/redis-db/browse_frm/thread/573cc5f5e3fcdf2b

[Josiah Carlson]

On Tue, Nov 15, 2011 at 4:37 AM, Salvatore Sanfilippo <ant...@gmail.com> wrote:
> Hi Ronny,
>
> before to continue actually replying your question, there are much
> better ways to handle your use case, especially the following:
>
> 1) Start a new instance of Redis.
> 2) Make the new instance a clone of your production instance using:
> SLAVEOF <host> <port>
> 3) When the replication finished cloning the instance (you can see
> that from the logs of the slave), turn the new instance into a slave
> with: SLAVEOF NO ONE.

Salvatore of course means "disable slaving and turn the slave into a
stand-alone master."

> 4) At this point manipulate the slave instance as you wish.
> 5) When you are ready finally either switch the client configuration
> to use the new instance, or, use replication again (SLAVEOF and so
> forth) in order to clone the new data into the old instance. But this
> will cause some downtime, so switching the instance is the way to go
> IMHO.
>
> make sure to do a backup of your most recent RDB file before doing all
> that. Even if you are using AOF persistency you can easily create an
> RDB backup calling the BGSAVE command.
>
> Note that while you alter the data on the new instance all the changes
> arriving on the old instance are not replicated since we disconnected
> the two instances with SLAVEOF NO ONE. But this also happened in your
> scenario with two databases.

Also note that even when you have a slave of the master, while writes
to the master will replicate to the slave, writes to the slave will
not replicate to the master.

Regards,
- Josiah

[Ronny López]

Thank you very much for your help guys.

It worked perfectly this way.

Anyway, I am still curious how to move all keys from one database to
another (in the same redis instance). So if you guys have some time,
would be good to know about the method Salvatore talked about.

Cheers,
Ronny.

On Nov 15, 5:16 pm, Josiah Carlson <josiah.carl...@gmail.com> wrote:

> > On Tue, Nov 15, 2011 at 1:07 PM, Ronny López <ronnyh...@gmail.com> wrote:
> >> Hello,
>
> >> We need to move all the keys in a Redis database to another. For
> >> example, from DB 0 to DB 1, in the same redis instance.
>
> >> We are upgrading the application, and we need to perform certain
> >> operations with  the data, but keep the application running. For this
> >> reason, we are going to perform the changes over a new database,
> >> cloned from the current one, and then change the application to use
> >> this new database.
>
> >> Another use case is unit testing. We need to clone the database to run
> >> the tests with the application data, but in a test database.
>
> >> So, how this can be done? What are the best practices?
>
> >> Thanks.
>
> >> --
> >> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> >> To post to this group, send email to redi...@googlegroups.com.
> >> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.

> >> For more options, visit this group athttp://groups.google.com/group/redis-db?hl=en.

[Josiah Carlson]

You can use the MOVE command to move keys to another database:
http://redis.io/commands/move

Regards,
- Josiah

[Andy McCurdy]

When we started using Redis, we only had a single instance. We had multiple apps talking to different DBs on that instance. As we grew, we needed to split that up. Now we have a policy that each app be in it's own instance, and we only utilize DB 0 on those instances.

We initially tried to do the conversion with the AOF log, but in our tests it would have taken nearly an hour to restore the DB from an AOF log. Reloading from RDB is *much* quicker, coming back online in under a minute.

After reading the Redis persistence code, I realized it was quite trivial to rewrite an RDB file replacing the DB that it "SELECTs".

Here's a link to the code I used to make the change. https://gist.github.com/1368496

Warning: This is exceptionally hacky. And there's some caveats:

- It requires a brief downtime. The code runs very quickl. I converted a 35G RDB file in under 30 seconds.

- The destination DB is hardcoded to 0. If you want something else, it's trivial to change though.

- It only converts the first SELECT statement to use the destination DB. If you have more than one DB currently in the RDB file, only the first one will get converted. The rest will remain untouched.

Given that, if this still meets your needs, have at it!

-andy

[Salvatore Sanfilippo]

Hi Andy,

thanks for sharing this, maybe we need rdb-check utility smarter than
it is and able to do this kind of magics?
Like merging DBs, splitting DBs, and possibly even looking for
specific keys and dump it.

Cheers,
Salvatore

[Dvir Volk]

slightly off topic, but related:

I'm actually now facing the need to distribute and update data from a backend data crunching machine, to a database that serves the users, without any down time, and constantly.

I don't mind if part of the data is inconsistent for a few minutes, as it is mostly scores for search terms, etc.

I thought of doing a script that does the following (python-ish pseudo code):

source = Redis(host = back, db = x)

dest = Redis( host =front, db = y )

while True:

   k = source.randomkey()

   if not k:

       break

   t = source.type(k)

   val = readVal(source, k, t) #I won't go into the details of this function ;)

   writeVal(dest, k, t, val) #

   

   source.delete(k)

this will be slow, although batching things up with pipelines will speed them up.

also, I will not be able to delete keys that don't exist in source but exist in dest, but that can be solved by taking snapshots of the keys on each transfer, keeping a set of all keys (to avoid the KEYS command), or doing a nightly "big sync" with SLAVEOF every day or two.

the other option is of course juggling instances of and on, but this can be a nice solution for this use case.

  

Dvir Volk

System Architect, DoAT, http://doat.com

[Nike]

The tool would be super useful. I found databases very useful for
quick development with further dividing into separate instances when
(if) they grow enough