RavenDB tanking itself in terms of RAM

[Craig Brett]

Hi there,

We're using RavenDB as part of our data import / processing side of things, as well as for display. We're running processes that will pull a document from Raven, process it and put it back in. Admittedly we can do this at pretty high speeds, we're using Windows Azure worker roles to do the heavy lifting before putting the data back.

What we're finding is even with indexing paused on the effected databases, the memory for RavenDB goes up and up and up until it reaches 99%, and then (understandably) it stops servicing responses and the database becomes unusable. Then the Azure worker roles get upset because they can't write to or read from Raven, and things just go south from there.

Is this because of sheer number of requests? We're not talking an unreasonable amount of incoming data here, definitely not enough to fill up the RAM with the documents alone at any one time.

On a probably related note, when we stop processing and restart the Raven service (we have to since it doesn't recover by itself), it will rightly start indexing stuff. This too leads to Raven committing memory suicide again.

What (if anything) are we doing wrong here? I'm happy to send server logs to someone looking into this, though I can't post it on this group as it contains the data as it starts being indexed again.

I have observed these exceptions occuring in the logs at around the time it all went weird and I'm posting them in case they help:

2014-08-14 14:25:10.3386,Raven.Database.Server.Abstractions.HttpListenerContextAdpater,Info,Failed to close response,"System.AggregateException: One or more errors occurred. ---> System.Net.HttpListenerException: The specified network name is no longer available
   at System.Net.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at Raven.Database.Util.Streams.BufferPoolStream.InternalFlush()
   at Raven.Database.Util.Streams.BufferPoolStream.Flush()
   at Raven.Database.Impl.ExceptionAggregator.Execute(Action action)
   --- End of inner exception stack trace ---
   at Raven.Database.Impl.ExceptionAggregator.ThrowIfNeeded()
   at Raven.Database.Server.Abstractions.HttpListenerResponseAdapter.Close()
   at Raven.Database.Server.Abstractions.HttpListenerContextAdpater.FinalizeResponse()
---> (Inner Exception #0) System.Net.HttpListenerException (0x80004005): The specified network name is no longer available
   at System.Net.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at Raven.Database.Util.Streams.BufferPoolStream.InternalFlush()
   at Raven.Database.Util.Streams.BufferPoolStream.Flush()
   at Raven.Database.Impl.ExceptionAggregator.Execute(Action action)<---
"

and in a non-aggregate flavour

2014-08-14 14:29:58.2577,Raven.Database.Server.HttpServer,Warn,Error on request,"System.Net.HttpListenerException (0x80004005): The specified network name is no longer available
   at System.Net.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at Raven.Database.Util.Streams.BufferPoolStream.InternalFlush()
   at Raven.Database.Util.Streams.BufferPoolStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Compression.DeflateStream.WriteDeflaterOutput(Boolean isAsync)
   at System.IO.Compression.DeflateStream.Write(Byte[] array, Int32 offset, Int32 count)
   at System.IO.Compression.GZipStream.Write(Byte[] array, Int32 offset, Int32 count)
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Write(Char[] buffer, Int32 index, Int32 count)
   at Raven.Imports.Newtonsoft.Json.Utilities.JavaScriptUtils.WriteEscapedJavaScriptString(TextWriter writer, String s, Char delimiter, Boolean appendDelimiters, Boolean[] charEscapeFlags, StringEscapeHandling stringEscapeHandling)
   at Raven.Imports.Newtonsoft.Json.JsonTextWriter.WriteValue(String value)
   at Raven.Json.Linq.RavenJValue.WriteTo(JsonWriter writer, JsonConverter[] converters)
   at Raven.Json.Linq.RavenJObject.WriteTo(JsonWriter writer, JsonConverter[] converters)
   at Raven.Json.Linq.RavenJArray.WriteTo(JsonWriter writer, JsonConverter[] converters)
   at Raven.Json.Linq.RavenJObject.WriteTo(JsonWriter writer, JsonConverter[] converters)
   at Raven.Json.Linq.RavenJArray.WriteTo(JsonWriter writer, JsonConverter[] converters)
   at Raven.Json.Linq.RavenJObject.WriteTo(JsonWriter writer, JsonConverter[] converters)
   at Raven.Database.Extensions.HttpExtensions.WriteJson(IHttpContext context, RavenJToken obj)
   at Raven.Database.Server.Responders.Index.GetIndexQueryResult(IHttpContext context, String index)
   at Raven.Database.Server.HttpServer.DispatchRequest(IHttpContext ctx)
   at Raven.Database.Server.HttpServer.HandleActualRequest(IHttpContext ctx)"

And slightly similarly:

2014-08-14 14:29:59.6774,Raven.Database.Server.Abstractions.HttpListenerContextAdpater,Info,Failed to close response,"System.AggregateException: One or more errors occurred. ---> System.Net.HttpListenerException: An operation was attempted on a nonexistent network connection
   at System.Net.HttpResponseStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at Raven.Database.Util.Streams.BufferPoolStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at Raven.Database.Impl.ExceptionAggregator.Execute(Action action)
   --- End of inner exception stack trace ---
   at Raven.Database.Impl.ExceptionAggregator.ThrowIfNeeded()
   at Raven.Database.Server.Abstractions.HttpListenerResponseAdapter.Close()
   at Raven.Database.Server.Abstractions.HttpListenerContextAdpater.FinalizeResponse()
---> (Inner Exception #0) System.Net.HttpListenerException (0x80004005): An operation was attempted on a nonexistent network connection
   at System.Net.HttpResponseStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at Raven.Database.Util.Streams.BufferPoolStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at Raven.Database.Impl.ExceptionAggregator.Execute(Action action)<---
"

This one happens pretty frequently during/after Raven tanks

2014-08-14 14:39:09.1043,Raven.Database.Server.HttpServer,Warn,Error on request,"System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowIfCancellationRequested()
   at Raven.Database.Indexing.Index.IndexQueryOperation.<Query>d__54.MoveNext()
   at Raven.Database.Util.ActiveEnumerable`1..ctor(IEnumerable`1 enumerable)
   at Raven.Database.DocumentDatabase.<>c__DisplayClass8f.<Query>b__84(IStorageActionsAccessor actions)
   at Raven.Storage.Esent.TransactionalStorage.ExecuteBatch(Action`1 action, EsentTransactionContext transactionContext)
   at Raven.Storage.Esent.TransactionalStorage.Batch(Action`1 action)
   at Raven.Database.DocumentDatabase.Query(String index, IndexQuery query, CancellationToken externalCancellationToken, Action`1 headerInfo, Action`1 onResult)
   at Raven.Database.DocumentDatabase.Query(String index, IndexQuery query, CancellationToken token)
   at Raven.Database.Server.Responders.Index.PerformQueryAgainstExistingIndex(IHttpContext context, String index, IndexQuery indexQuery, CancellationToken token, Etag& indexEtag)
   at Raven.Database.Server.Responders.Index.ExecuteQuery(IHttpContext context, String index, CancellationToken token, Etag& indexEtag)
   at Raven.Database.Server.Responders.Index.GetIndexQueryResult(IHttpContext context, String index)
   at Raven.Database.Server.HttpServer.DispatchRequest(IHttpContext ctx)
   at Raven.Database.Server.HttpServer.HandleActualRequest(IHttpContext ctx)"

We're using server build number 2879 and running Raven as a service.

Hopefully that's enough starting info to make it clear to someone in the know.

Any help appreciated.

Cheers,
Craig

[Mauro Servienti]

What build?
How is RavenDB hosted in Azure?

.m

Sent from my Amazing Yellow Lumia, typos are guaranteed ;-)

---

From: Craig Brett
Sent: ‎15/‎08/‎2014 15:18
To: rav...@googlegroups.com
Subject: [RavenDB] RavenDB tanking itself in terms of RAM

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Craig Brett]

Its build number 2879.

And we don't have Raven running in Azure. We're running it on-premises. The processing happens in Azure.

[Chris Marisic]

System.Net.HttpListenerException (0x80004005): The specified network name is no longer available

Almost sounds like you're reaching socket exhaustion from the TCP stack itself. Are you generating THOUSANDS or TENS OF THOUSANDS requests per second?

[Oren Eini (Ayende Rahien)]

Also, are you using Compression bundle?

What is the size of the documents? How many indexes? What are you doing in the indexes?

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-622-7811

Fax:      + 972-153-4622-7811

On Fri, Aug 15, 2014 at 6:44 PM, Mauro Servienti <ma...@topics.it> wrote:

What build?
How is RavenDB hosted in Azure?

.m

Sent from my Amazing Yellow Lumia, typos are guaranteed ;-)

---

From: Craig Brett
Sent: 15/08/2014 15:18

[Craig Brett]

No, well I'd be surprised. When we're processing, it may be hundreds of reads/writes per second, but we're not running with enough capacity to reach tens of thousands per second. If we did, would it be up to us to monitor RavenDB's use of the TCP stack? I wouldn't know how even if that was the case. Might be worth including it in the docs if this is something other devs might get caught by.

Raven is still being awkward even now, filling itself on memory and playing dead after a while. The only solution on a temporary basis is to restart the service until it does it again. I haven't processed any more stuff since about last Thursday.

[Craig Brett]

The documents range in size from small ones to ones at around about 1MB. Probably more, but no particularly large docs spring to mind to check.

We have 68 indexes spread across 5 databases. We've got 3.8m documents in these databases too. The indexes are mostly maps, but there are several map reduces (maybe about 5 at a guess).

No, it doesn't look like we do have the compression bundle there. Should we?

[Mauro Servienti]

have you got indexes with LoadDocument?

.m

--

(no keyboard keys have been killed due to the really annoying OSX spell checker)

[Craig Brett]

Our biggest index does, yes. It loads at most 2 small small documents per entry. There are only 150k documents in this index though.

Could that be a culprate?

[Mauro Servienti]

my experience says yes, we have faced the exact same pain (especially on 2.0) and everything was solved removing all the “LoadDocument” in the indexes.

Sent from Windows Mail

[Craig Brett]

It seems strange that that would be a problem though with indexing paused on those databases. I'd be surprised if it was doing any LoadDocument stuff for indexes with indexing paused. I'm not saying it doesn't, I'd just be surprised.

[Chris Marisic]

When you use LoadDocument in an index, RavenDB needs to store (effectively) relationship tables. Otherwise it would not know that you modified User document and it needs to modify AccountIndex.

When you pause indexing, I can't imagine that it can pause the relationship management as I don't see how it would recover that information. The relationship tables also need to be transactionally consistent during writes which extends the breadth of transactions for writes.

Oren is that a fair assessment of how load document operates?

[Oren Eini (Ayende Rahien)]

Pretty much, yes. 

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-622-7811

Fax:      + 972-153-4622-7811

--

[Craig Brett]

Ok, we're going to give removing LoadDocument a try and using a hardcoded value instead.

The documents that were being loaded were really small, but I guess that can add up, so we'll see how this goes. Thanks for the heads up.

[Chris Marisic]

Don't forget you can always maintain denormalized relational data if the cardinality and rate of change makes sense. Suppose you have Users and Accounts and you want to search by FirstName on both. You can use load document as 1 solution, or in the rare situation that a person changes their name you can just update both User and Account at the same time. As long as you plan well, it all can work just fine.

I can even say with experience that I've worked with systems that has issues with duplication (what system usually doesn't at some point?). Users created multiple accounts for themselves and then eventually all the stuff needs merged into 1 account. I've rekeyed every single document in the database tied to a person before, Raven actually made it really awesome to do so. In our case we merely added UserId to the DocumentsByName index and then i could soak up the entire database for UserId:1 and modify the json and just re-persist every document related to that person. Is that something to do all the time? Of course not. Is that a good solution to something as infrequent and complicated as fixing duplicate account issues? Yes.

[Craig Brett]

Ok, well, it was met with partial success! At least while just indexing while no heavy duty processing is going on, Raven's memory seems to be stable. Which is a big improvement, as now we can just stop throwing new data at it when we need it to be definitely online.

When we are processing, we are still (though seemingly less frequently) getting these issues. It could be that there's other load document stuff going on somewhere, but I'm not aware of it. We're also guessing that some of this is the fault of hardware, as we're running this on a repurposed desktop machine (still got 12GB RAM but still), so we're going to try it in a proper server to see if that helps.

For now, I'm just processing the data slower, to give Raven a chance to catch up with all the stuff we're throwing at it. But removing LoadDocument has helped. Thanks! Denormalized data may well work here and is something we're thinking about as an alternative.

[Mircea Chirea]

FIY server hardware is just more reliable desktop hardware, with more management features (except storage drives).

[Chris Marisic]

On Wednesday, August 27, 2014 9:44:44 AM UTC-4, Craig Brett wrote:

 We're also guessing that some of this is the fault of hardware, as we're running this on a repurposed desktop machine (still got 12GB RAM but still), so we're going to try it in a proper server to see if that helps.

Disk performance is pretty important for Raven, SSDs are best for the fast nonsequential reads. Most data access in a nonrelational store is nonsequential, rotational platters are good for sequential access.

[Oren Eini (Ayende Rahien)]

Most data access in most databases are non sequential.

Note that pretty much only OLAP does big seqential stuff.

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-622-7811

Fax:      + 972-153-4622-7811

--