Billions of files on SeaweedFS filer?

[horizon infinity]

Hi all,

Anyone out there has previous experience storing a large number of files (say a billion or more) on filer?

We have a project that requires us to store billions of small object files. After evaluating several tools, we felt SeaweedFS is the right one for accomplishing this because of its design.

There is a catch though.  We stumbled on the FID mapping to the real object stored on SeaweedFS.  To maintain billions of keys that map to SeaweedFS FID is a challenge itself as our keys are fairly complicated.  While we are still searching the appropriate storage layer for storing keys, I recall that SeaweeFS comes with filer support.  This may work for us as our key is consisted of path pattern, e.g. CatA/CatB/CatC/CatD/CatE/CatF ....

Before I set out to test the filer function, I would like seek the similar experience and things to watch out for when using filer to handle such a large number of files.

Or if you have managed to maintain the original file / FID mapping using certain storage engines and if you are willing to shed some lights on how it can be done efficiently, I will love to learn from your sharing.

Cheers,

Paul

[Chris Lu]

It really depends on your directory structure. Do you have a few "Cat X", or deep path, or shallow path, etc.

SeaweedFS' Filer default implementation is to put directory structure in memory, each level is a map. The leaf files are dir_id~fid mapping in on-disk leveldb.

The cassandra storage engine could be good for your case also, since it can be distributed easily.

Chris

--
You received this message because you are subscribed to the Google Groups "Seaweed File System" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.