OutOfMemoryException with streaming data

[Shawn Mclean]

We have the following code snippet that tries to stream a collection of documents that are about 5mb in size:

var query = RavenSession.Query<dynamic>("SetIndex");
     
using (var result = RavenSession.Advanced.Stream(query))
{
  while (result.MoveNext())
  {
  }
}

This throws an out of memory at `result.MoveNext()` when we have reached iteration 14. Is this underlying implementation adding the data to the results or does it flush it? If thats the case, how do we clear previous documents from results?

Our stacktrace:

System.AggregateException : One or more errors occurred.
  ----> System.OutOfMemoryException : Exception of type 'System.OutOfMemoryException' was thrown.

   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Raven.Client.Embedded.EmbeddedDatabaseCommands.<YieldUntilDone>d__32.<>m__Finally34() in c:\Builds\RavenDB-Stable-2.5\Raven.Client.Embedded\EmbeddedDatabaseCommands.cs:line 642
   at Raven.Client.Embedded.EmbeddedDatabaseCommands.<YieldUntilDone>d__32.System.IDisposable.Dispose() in c:\Builds\RavenDB-Stable-2.5\Raven.Client.Embedded\EmbeddedDatabaseCommands.cs:line 0
   at Raven.Client.Document.DocumentSession.<YieldQuery>d__c`1.System.IDisposable.Dispose() in c:\Builds\RavenDB-Stable-2.5\Raven.Client.Lightweight\Document\DocumentSession.cs:line 0
   at Raven.Client.Document.DocumentSession.<YieldQuery>d__c`1.MoveNext() in c:\Builds\RavenDB-Stable-2.5\Raven.Client.Lightweight\Document\DocumentSession.cs:line 606
   at Specs.EntityExporters.WhenIExportAnEntity.DumpStuff_Temp() in C:\grc\core\Specs\EntityExporters\WhenIExportAnEntity.cs:line 30
--OutOfMemoryException
   at System.String.CtorCharArrayStartLength(Char[] value, Int32 startIndex, Int32 length)
   at Raven.Imports.Newtonsoft.Json.Bson.BsonReader.ReadString() in c:\Builds\RavenDB-Stable-2.5\Imports\Newtonsoft.Json\Src\Newtonsoft.Json\Bson\BsonReader.cs:line 648
   at Raven.Imports.Newtonsoft.Json.Bson.BsonReader.ReadNormal() in c:\Builds\RavenDB-Stable-2.5\Imports\Newtonsoft.Json\Src\Newtonsoft.Json\Bson\BsonReader.cs:line 439
   at Raven.Imports.Newtonsoft.Json.Bson.BsonReader.ReadInternal() in c:\Builds\RavenDB-Stable-2.5\Imports\Newtonsoft.Json\Src\Newtonsoft.Json\Bson\BsonReader.cs:line 257
   at Raven.Imports.Newtonsoft.Json.Bson.BsonReader.Read() in c:\Builds\RavenDB-Stable-2.5\Imports\Newtonsoft.Json\Src\Newtonsoft.Json\Bson\BsonReader.cs:line 245
   at Raven.Json.Linq.RavenJArray.Load(JsonReader reader) in c:\Builds\RavenDB-Stable-2.5\Raven.Abstractions\Json\Linq\RavenJArray.cs:line 151
   at Raven.Json.Linq.RavenJObject.Load(JsonReader reader) in c:\Builds\RavenDB-Stable-2.5\Raven.Abstractions\Json\Linq\RavenJObject.cs:line 220
   at Raven.Abstractions.Extensions.JsonExtensions.ToJObject(Stream self) in c:\Builds\RavenDB-Stable-2.5\Raven.Abstractions\Extensions\JsonExtensions.cs:line 50
   at Raven.Storage.Managed.DocumentsStorageActions.ReadDocument(Tuple`3 stream, JsonDocumentMetadata metadata) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Storage\Managed\DocumentsStorageActions.cs:line 234
   at Raven.Storage.Managed.DocumentsStorageActions.<DocumentByKey>b__14(Tuple`3 tuple, JsonDocumentMetadata metadata) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Storage\Managed\DocumentsStorageActions.cs:line 134
   at Raven.Storage.Managed.DocumentsStorageActions.DocumentByKeyInternal[T](String key, TransactionInformation transactionInformation, Func`3 createResult) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Storage\Managed\DocumentsStorageActions.cs:line 196
   at Raven.Storage.Managed.DocumentsStorageActions.DocumentByKey(String key, TransactionInformation transactionInformation) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Storage\Managed\DocumentsStorageActions.cs:line 134
   at Raven.Database.Impl.DocumentRetriever.GetDocumentWithCaching(String key) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Impl\DocumentRetriever.cs:line 214
   at Raven.Database.Impl.DocumentRetriever.RetrieveDocumentInternal(IndexQueryResult queryResult, HashSet`1 loadedIds, FieldsToFetch fieldsToFetch, IndexDefinition indexDefinition) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Impl\DocumentRetriever.cs:line 117
   at Raven.Database.Impl.DocumentRetriever.ShouldIncludeResultInQuery(IndexQueryResult arg, IndexDefinition indexDefinition, FieldsToFetch fieldsToFetch) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Impl\DocumentRetriever.cs:line 246
   at Raven.Database.DocumentDatabase.<>c__DisplayClass97.<>c__DisplayClass9b.<Query>b__8d(IndexQueryResult result) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\DocumentDatabase.cs:line 1374
   at Raven.Database.Indexing.Index.IndexQueryOperation.ShouldIncludeInResults(IndexQueryResult indexQueryResult) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\Index.cs:line 1189
   at Raven.Database.Indexing.Index.IndexQueryOperation.<Query>d__57.MoveNext() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Indexing\Index.cs:line 1018
   at Raven.Database.Util.ActiveEnumerable`1.MoveNext() in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Util\ActiveEnumerable.cs:line 46
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at Raven.Database.DocumentDatabase.<>c__DisplayClass97.<Query>b__8c(IStorageActionsAccessor actions) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\DocumentDatabase.cs:line 1401
   at Raven.Storage.Managed.TransactionalStorage.ExecuteBatch(Action`1 action) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Storage\Managed\TransactionalStorage.cs:line 160
   at Raven.Storage.Managed.TransactionalStorage.Batch(Action`1 action) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\Storage\Managed\TransactionalStorage.cs:line 139
   at Raven.Database.DocumentDatabase.Query(String index, IndexQuery query, CancellationToken externalCancellationToken, Action`1 headerInfo, Action`1 onResult) in c:\Builds\RavenDB-Stable-2.5\Raven.Database\DocumentDatabase.cs:line 1345
   at Raven.Client.Embedded.EmbeddedDatabaseCommands.<>c__DisplayClass27.<StreamQuery>b__21() in c:\Builds\RavenDB-Stable-2.5\Raven.Client.Embedded\EmbeddedDatabaseCommands.cs:line 557
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()

[Oren Eini (Ayende Rahien)]

You are using embedded version?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shawn Mclean]

Yes, testing on embedded database. Our prod uses the server.

[Oren Eini (Ayende Rahien)]

And you use 32 bits?

The problem is that you are paying both costs, server and client, and the same app, and you are running out of address space.

[Shawn Mclean]

Yes, the application is 32-bits, we can't upgrade to 64-bits due to some dependencies. So you're saying the same issue would apply even if we're running on the server once the app is on 32-bit?

[Oren Eini (Ayende Rahien)]

No, run the client and server as separate processes, it will work