Write parallelism for multiple leveldb instances (each on different disk) with threads

[ashokana]

We have a case of single process, opened multiple leveldb instances , one leveldb instance per drive, and trying to write to these 3 instances with 3 threads. The expectation was that if on single drive, we get 100 MBps - we should get 300 MBps. We do see such behaviour on read - but on write - we see performance same as that of single instance even using 3 threads. All these leveldb instances use same env, i.e., default one. Any thoughts on why the writes would not give 3X performance for leveldb.

- Could singleton default env be causing issue here?

- We hacked in leveldb code to return different env for each thread and that seems to give close to 3X even for writes. Any thoughts here?

[Sanjay Ghemawat]

The default Env has just a single thread that it uses to handle all background work.  So your three leveldb instances are all sharing the same thread to do their background work (mainly compactions).

I think this should be fixed in the default env so it uses separate threads for separate Schedule() calls.  It might be necessary to add in a pool of idle threads so that we don't end up creating/destroying a brand new thread for every piece of background work (depends on how fast thread creation/destruction is).

--
You received this message because you are subscribed to the Google Groups "leveldb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leveldb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ashok Anand]

we could extend EnvWrapper's schedule API to schedule task in a separate background thread (instead of using Posix env schedule) and use different instances of envwrapper for our purpose..?

To unsubscribe from this group and stop receiving emails from it, send an email to leveldb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "leveldb" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/leveldb/0BgX-eA8FX8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to leveldb+unsubscribe@googlegroups.com.

[Sanjay Ghemawat]

On Wed, Feb 7, 2018 at 3:47 PM Ashok Anand <ashok...@gmail.com> wrote:

we could extend EnvWrapper's schedule API to schedule task in a separate background thread (instead of using Posix env schedule) and use different instances of envwrapper for our purpose..?

Yes, that would work for your needs.  Though you won't need different EnvWrappers necessarily.  You could instead have a single EnvWrapper whose Schedule() creates a new pthread for each call.

When I wrote what I did, I was thinking out loud about how leveldb should behave by default, so consider what I wrote something for the leveldb team to work on; you can use an EnvWrapper to work around the problem until leveldb has been fixed.

 

On Wed, Feb 7, 2018 at 11:31 AM, 'Sanjay Ghemawat' via leveldb <lev...@googlegroups.com> wrote:

The default Env has just a single thread that it uses to handle all background work.  So your three leveldb instances are all sharing the same thread to do their background work (mainly compactions).

I think this should be fixed in the default env so it uses separate threads for separate Schedule() calls.  It might be necessary to add in a pool of idle threads so that we don't end up creating/destroying a brand new thread for every piece of background work (depends on how fast thread creation/destruction is).

On Wed, Feb 7, 2018 at 9:23 AM ashokana <ashok...@gmail.com> wrote:

We have a case of single process, opened multiple leveldb instances , one leveldb instance per drive, and trying to write to these 3 instances with 3 threads. The expectation was that if on single drive, we get 100 MBps - we should get 300 MBps. We do see such behaviour on read - but on write - we see performance same as that of single instance even using 3 threads. All these leveldb instances use same env, i.e., default one. Any thoughts on why the writes would not give 3X performance for leveldb.

- Could singleton default env be causing issue here?

- We hacked in leveldb code to return different env for each thread and that seems to give close to 3X even for writes. Any thoughts here?

--
You received this message because you are subscribed to the Google Groups "leveldb" group.

To unsubscribe from this group and stop receiving emails from it, send an email to leveldb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "leveldb" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/leveldb/0BgX-eA8FX8/unsubscribe.

To unsubscribe from this group and all its topics, send an email to leveldb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "leveldb" group.

To unsubscribe from this group and stop receiving emails from it, send an email to leveldb+u...@googlegroups.com.