Developing RocksDB multi-store server

[Jan Lisowiec]

Hi

I've developed a RocksDB server with a layer that aggregates multiple instances of RocksDB databases within a single address space. For the client front-end I've developed a unified API that can talk to each instance via a dedicated high-performance protocol. Since the server creates and manages multiple instances of RocksDB, potentially a large number, I need a way to manage file descriptors and memory across all these instances. RocksDB allows putting limits on these resources per individual database. The question is, has anyone developed a way to control these resources across multiple instances running within the same address space?

Best

Jan Lisowiec

[Adam Retter]

Have you looked at the Env class which can be used across multiple RocksDB?

> --
> You received this message because you are subscribed to the Google Groups
> "rocksdb" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to rocksdb+u...@googlegroups.com.
> To post to this group, send email to roc...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/rocksdb/bcae67ca-8261-49d5-a4c1-a22e43b130ae%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Adam Retter

skype: adam.retter
tweet: adamretter
http://www.adamretter.org.uk

[Igor Canadi]

Hi Jan,

Unfortunately I don't think you can set one file descriptor limit for multiple databases - it will need to be per-database. However, as Adam says, the databases should share both the Env object and block_cache, which will share the thread pool and the userspace cache. This is how many services at Facebook run RocksDB -- multiple databases in a single process.

Igor

To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/CAPK0iciqy2HJRN4EMf9HAEq0v5bfAALR6kcBzGW5pqJ6x_W9Hw%40mail.gmail.com.

[Jan Lisowiec]

Hi Igor,

Thanks for the tip. 

I'm also trying to integrate it with Ceph. I can run it on top of CephFS but I guess it should be also possible to integrate it with Librado. I'd be interested in knowing if Env provides enough functionality so an adaptation to Librado can be developed?

Best

Jan

also 

[Igor Canadi]

Yes, I'm pretty sure you can integrate it with Ceph. At Facebook we integrated it with HDFS (https://github.com/facebook/rocksdb/blob/master/hdfs/env_hdfs.h) and now at Rockset we integrated with S3 (check it out: https://github.com/rockset/rocksdb-cloud). If you end up implementing Ceph Env, we would love your contribution to RocksDB cloud project.

Best,

Igor

To unsubscribe from this group and stop receiving emails from it, send an email to rocksdb+unsubscribe@googlegroups.com.

To post to this group, send email to roc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/50a5d9f8-c12c-4258-b01f-ba62aa6d4ca0%40googlegroups.com.

[MARK CALLAGHAN]

Isn't there work elsewhere on RocksDB + Ceph?
https://www.google.com/search?q=rocksdb+ceph&oq=rocksdb+ceph

To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/CAPUde8owBMfPaDiRyEejxuPPS0qLm2ymncE%2B7nq9T_-hEBOt6A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Mark Callaghan
mdca...@gmail.com

[Igor Canadi]

Oh yeah, looks like it's already there in the upstream: https://github.com/facebook/rocksdb/blob/master/utilities/env_librados.md

To unsubscribe from this group and stop receiving emails from it, send an email to rocksdb+u...@googlegroups.com.

To post to this group, send email to roc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/50a5d9f8-c12c-4258-b01f-ba62aa6d4ca0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rocksdb" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rocksdb+u...@googlegroups.com.

To post to this group, send email to roc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/CAPUde8owBMfPaDiRyEejxuPPS0qLm2ymncE%2B7nq9T_-hEBOt6A%40mail.gmail.com.