AOF or DUMP splitting

[Goldbach]

I have a fairly big AOF to migrate to a clustered architecture. I am currently using this approach

 - N-1 nodes in the cluster are empty

 - the Nth has all data

As soon as you load the data in the Nth, Redis realizes the node has keys he should not have, and marks them for import; then redis-trib.fix does the redistribution magic.

Anyway it takes time and consumes a lot of bandwidth

If I split the AOF in N parts, and restart all redis instances, do I gain anything? Either in time or network traffic saved?

If yes, is there a reliable way to split an AOF in parts? N is reasonably small, so, if I have to do some manual work on the tail and head of each split, it is fine.

[Josiah Carlson]

The AOF is a sequential log file. It says what operations were performed on what keys and in what order. Splitting up the AOF would have different machines operating on partial data simultaneously, operating on keys that don't exist on the machine, and I can just about guarantee will lead to data corruption. So don't do that.

There are various tools for splitting up .rdb snapshots, one of them being redis-rdb-tools. If you've got an rdb snapshot, using one of the rdb splitters may be a decent option.

Honestly, unless you're paying per byte in bandwidth, letting Redis cluster fix itself might be the easiest option. Alternatively, if you can let your old machine stay up, there are several copy/migrate tools that can migrate your old data key by key, or using the redis-rdb-tools, you can read the operations straight out of the rdb and write them to your cluster key by key.

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.