Munin Questions

[CollegeDEV]

I've posted in a few other threads here, and I'm sort of new to the RavenDB community.  I'm wanting to use a .NET-based NoSQL implementation for really fast lookups and full-text searching to power several jQuery autocomplete search boxes that have a 1+ million rows of data.  Raven seemed perfect for my needs.  I've been using Munin because the response times are often 1 or 2ms, whereas ESent is several hundred milliseconds for the same query; I'm using normal spindle hard drives, nothing fancy.  Here are my questions in regards to Munin:

1.  I've seen where Munin will "die" if idle for extensive periods of time, such as overnight.  Is this garbage collection by the CLR?  Is there someway to prevent this?

2.  Is there a way turn off logging to improve the performance?

3.  I saw Ayende's blog about upcoming compression, could this be available for Munin?  This could be sort of like xVelocity with Microsoft or Cassandra.

4.  Is Munin "safe" for Production use? Or was it intended to be only used for unit testing?

5.  Bonus Question:  Is Raven extensible enough so that if I wrote a C++ in-memory engine that it could easily plug-in to the existing system? (and theoretically be faster)

[Michael Weber]

Ayende has talked about Munin before here on the forums.  It is not production ready and has known race conditions.  Also, it will almost certainly be removed from the next version of RavenDB.

There is a replacement that he started, but isn't feature complete.  The branch is available at  https://github.com/ayende/ravendb/tree/ram 

You can certainly create another in-memory storage engine, although I don't see why writing it in C++ would be any faster.

[Oren Eini (Ayende Rahien)]

inline

On Tue, Aug 28, 2012 at 6:07 PM, CollegeDEV <tpo...@gmail.com> wrote:

I've posted in a few other threads here, and I'm sort of new to the RavenDB community.  I'm wanting to use a .NET-based NoSQL implementation for really fast lookups and full-text searching to power several jQuery autocomplete search boxes that have a 1+ million rows of data.  Raven seemed perfect for my needs.  I've been using Munin because the response times are often 1 or 2ms, whereas ESent is several hundred milliseconds for the same query;

This seems to be VERY surprising. We usually see single digit ms response times with Esent.

 

I'm using normal spindle hard drives, nothing fancy.  Here are my questions in regards to Munin:

1.  I've seen where Munin will "die" if idle for extensive periods of time, such as overnight.  Is this garbage collection by the CLR?  Is there someway to prevent this?

No, Munin isn't really suitable for production.

I am not sure what this is, but please note that Munin is likely to be replaced with a pure in memory storage for the next release.

 

2.  Is there a way turn off logging to improve the performance?

Standard NLog options, yes.

 

3.  I saw Ayende's blog about upcoming compression, could this be available for Munin?  This could be sort of like xVelocity with Microsoft or Cassandra.

It is at a layer on top of the storage, so yes.

 

4.  Is Munin "safe" for Production use? Or was it intended to be only used for unit testing?

No, Yes.

 

5.  Bonus Question:  Is Raven extensible enough so that if I wrote a C++ in-memory engine that it could easily plug-in to the existing system? (and theoretically be faster)

You could do that, sure. ITransactionalStorage is the entry point.

But I doubt that you could really get it to be much faster.

[Chris Marisic]

On Tuesday, August 28, 2012 12:09:58 PM UTC-4, Oren Eini wrote:

inline

On Tue, Aug 28, 2012 at 6:07 PM, CollegeDEV <tpo...@gmail.com> wrote:

I've posted in a few other threads here, and I'm sort of new to the RavenDB community. 

 

5.  Bonus Question:  Is Raven extensible enough so that if I wrote a C++ in-memory engine that it could easily plug-in to the existing system? (and theoretically be faster)

You could do that, sure. ITransactionalStorage is the entry point.

But I doubt that you could really get it to be much faster.

Once again I'm making this comment based off your username, you are not capable of doing a better job at building a storage engine than Oren is. Not only does Oren have over atleast a decade of experience in ultra complex system software development RavenDB has been his baby now for over 2 years.

Microsoft has Oren consult for them.

Do you actually believe your programming skills at this point are able to rival, let alone exceed Oren's?

Suppose somehow this even was the case, that you could create a storage system that theoretically shaves off a few milliseconds compared to RavenDB's usage of esent (likely after years of work), what exactly do you have to show for it? Next to nothing.

Software isn't about solving problems that have already been solved 3% faster, that's computer science. Computer science just like almost all pure science is only academic of value in nature.

Now supposing you even succeed at "faster" you'd likely be faster by losing true transaction isolation and instead replace always consistent with sometimes random things entirely go wrong, fast!

[Chris Marisic]

One last aspect to put in perspective. Oren didn't build ESENT. He chose to build RavenDB on top of a technology that has withstood the test of time and is shipped in every single major version of windows in the past decade. ESENT was likely created through the work of numerous teams over decades. There clearly is a reason Oren chose a tried and proven technology instead of attempting to reinvent the wheel.

[Oren Eini (Ayende Rahien)]

Hi,

Building an in memory store for RavenDB should be a fairly simple thing to do.

You need to implement the ITransactionalStorage interface, and that is about it.

If you would like to take a stab at it, I would be more than happy to accept a pull request.

As you mentioned, it probably doesn't make a lot of sense to try to create it in C++, because of interop costs.

On Thu, Aug 30, 2012 at 6:44 AM, CollegeDEV <tpo...@gmail.com> wrote:

I'm not IN College, I'm a college professor in addition to consulting at a major Wall Street investment bank.  No need for the computer science lessons.  In my world of finance, shaving a few milliseconds off a transaction is the difference between making money and losing money. I'm not knocking ESENT at all, I was just looking for a really fast in-memory NoSQL solution with full-text searching capabilities.  I want to keep it .NET-based since that's the target platform for this analytics engine, so thats why I asked about Munin.  Sure I could go with pure C++ based solutions but the UI aspects of it being "pretty" with WPF is just as important as speed.  There's going to be a lot of interop overhead on either side of C++ (either using a C++ engine with WPF on top or writing a C++ engine for Raven).

Please don't take my questions in the form of me thinking my skills are better (or worse) than Oren's.  My domain problem is unique in nature, and so I'm attempting to squeeze every single millisecond out of a .NET-based solution because the development effort to go to C++ is much higher and harder.  We're all professionals--let's keep it that way.  Don't worry, I believe we will be choosing RavenDB, and probably order our licenses here in the next few days.  RavenDB is a great product with a great future ahead of it.

[bbqchickenrobot]

Was your email even necessary or helpful in any way?  Who are you to assess the OP's skills and whether or not they are better than Oren's? Maybe they are - maybe the aren't... the real point here is - it doesn't matter and to do so shows that you're post came from ignorance. The cool thing about software is that you can do what you want with it if allowed.  And obviously Oren has baked in the ability to build others types of stores.  Personally, I like to see addins like this.  I hate to clutter this post even more but I just felt I had to say something.  

[Oren Eini (Ayende Rahien)]

If there a point in bringing something from 4 - 5 ago?

[bbqchickenrobot]

Yes. For Chris to understand that his post was useless and it only wastes other members/users time.  I'm trying to find solutions for alternative stores (since Munin was dropped) and I don't want to waste time on posts that don't actually contribute to the OPs question or topic.  It annoying when that happens.

[CollegeDEV]

bbqchicken,  FWIW, I've began testing using esent on a RAMDISK. You can basically stripe this RAID-style using sharding/replication.  I'm using DataRam's RAMDisk.  You can use something like HDTune to benchmark.  Random Access is typically what I look at.  Ramdisk is about 100x faster from a random access perspective than RAID-10 enterprise-grade 15k RPM spinning disks, and about 10x faster than a Samsung 840 Pro SSD.  But the SSD gets about 3x more IOPS.

But this is all still in a dev environment.  It has theoretically the same random latency as a Fusion-IO drive, but without the $10k cost.  Luckily our database is fairly small, only about 1.5GB.

[Chris Marisic]

The correct answer is to not reinvent the wheel. Creating a storage engine that needs to work 100% of the time, always, and forever is ludicrously hard, if not outright impossible except with multiple years of commitment.  So my answer does have meaning and is relevant. Creating your own storage engine is a guaranteed failure. There's a reason RavenDB is built to use ESENT, a storage engine that has been in production usage for over a decade.

If your goal is RavenDB usage outside of Windows Server, and using mono, it seems that the BerkleyDB project is thought to be the future for that. Working on that goal, as opposed to an infinite work project of creating your own engine would be much more meaningful, and also you might get some support from members of this group. Starting a storage engine will just result in abandonware because it's too large of a project. You will spend nearly your entire time implementing a working storage engine and never delivering business value.

[Matt Johnson]

+100 for BDB.

Remember, ESENT is high performance C/C++ code baked into windows.  Munin's attempt to do it in managed code is admirable, but if the concern is getting Linux support - then BDB would be a good parallel technology.  Who knows - it could perf test higher than ESENT and become the standard even on Windows for Raven 3.0.  (or whenever)

-1000 (negative) for trying to hand-write a custom datastore just for Raven.  One of the main ways I can sell people on Raven today is that it "uses the same backing technology as Active Directory and Exchange Server".  I'm sure there are equally impressive products based on BDB.