filer leveldb to redis?

56 views
Skip to first unread message

Andy Chen

unread,
Apr 25, 2018, 12:33:08 AM4/25/18
to Seaweed File System
hi, chris:
      now i run weed filer server with leveldb, how change to redis storage?Automatic conversion? or any method?

Chris Lu

unread,
Apr 25, 2018, 12:37:47 AM4/25/18
to seaw...@googlegroups.com
The storage is different and not easily interchangeable.

Btw: why switch to Redis?

Chris
On Tue, Apr 24, 2018 at 9:33 PM Andy Chen <nine5...@gmail.com> wrote:
hi, chris:
      now i run weed filer server with leveldb, how change to redis storage?Automatic conversion? or any method?

--
You received this message because you are subscribed to the Google Groups "Seaweed File System" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Andy Chen

unread,
Apr 25, 2018, 2:30:58 AM4/25/18
to Seaweed File System
Because leveldb has a single point of failure, I want to switch to redis as the master-slave backup.
The program I can think of:
1. Export the file list (weed export).
2. Then it is stored in redis (Analyze the file name and storage path). 
3. Restart filer server with redis.

I have seen that when using redis storage, is the file path used as the key(eg: /5ec14fe2d3c91325c71a8f0786c9e826), in the actual location of the volume as the value(eg: 3,06f33770e2), is not it? This program is feasible? Number of files already have 200 million

Chris Lu

unread,
Apr 25, 2018, 2:48:01 AM4/25/18
to Seaweed File System
weed export is iterating the files in one volume.  What you need is to iterate through the leveldb and persist each entry to redis.

The redis is a flat-namespace store, different from the level db implementation.

In leveldb, there is an auto-incremented directory id, and:
/*
The entry in level db has this format:
key: genKey(dirId, fileName)
value: []byte(fid)
And genKey(dirId, fileName) use first 4 bytes to store dirId, and rest for fileName
*/

It'll be good to know your use case, and how many directories are there.

Chris


To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+unsubscribe@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages