Build 2750 OrderBy throws OutOfMemoryException

90 views
Skip to first unread message

Mouhong Lin

unread,
Feb 27, 2014, 11:24:45 PM2/27/14
to rav...@googlegroups.com
Hi,

I'm using RavenDB build 2750. I reported "OrderBy is ignored by raven" a while ago in this thread:

Finally I use lucene query to solve that problem. But as the document number grows, now I get OutOfMemoryException.

The Model:

    public class ItemSimilarity
    {
        public string Id { get; set; }

        public string Item1Id { get; set; }

        public string Item2Id { get; set; }

        public string Type { get; set; }

        public double Similarity { get; set; }
    }

The Index:

    public class ItemSimilarityIndex : AbstractIndexCreationTask<ItemSimilarity>
    {
        public ItemSimilarityIndex()
        {
            Map = similarities => from sim in similarities
                                  select new
                                  {
                                      sim.Item1Id,
                                      sim.Item2Id,
                                      sim.Similarity,
                                      sim.Type
                                  };

            Index(x => x.Item1Id, Raven.Abstractions.Indexing.FieldIndexing.NotAnalyzed);
            Index(x => x.Item2Id, Raven.Abstractions.Indexing.FieldIndexing.NotAnalyzed);
            Index(x => x.Similarity, Raven.Abstractions.Indexing.FieldIndexing.NotAnalyzed);
            Index(x => x.Type, Raven.Abstractions.Indexing.FieldIndexing.NotAnalyzed);

            Sort(x => x.Similarity, Raven.Abstractions.Indexing.SortOptions.Double);
        }
    }


I have around 23,200,000 ItemSimilariy documents.

Now doing the query:

http://.../indexes/ItemSimilarityIndex?&query=Item1Id=686 AND Type=PageView&pageSize=128

The above query works fine, giving the result very quickly. But if I add the "sort":

http://.../indexes/ItemSimilarityIndex?&query=Item1Id=686 AND Type=PageView&pageSize=128&sort=-Similarity

It gave me OutOfMemoryException:

Url: \"/indexes/ItemSimilarityIndex?&query=Item1Id%3A1051%20AND%20Type%3APageView&pageSize=11&sort=-Similarity\"\r\n\r\nSystem.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at Lucene.Net.Search.FieldCacheImpl.DoubleCache.CreateValue(IndexReader reader, Entry entryKey)
   at Lucene.Net.Search.FieldCacheImpl.Cache.Get(IndexReader reader, Entry key)
   at Lucene.Net.Search.FieldCacheImpl.GetDoubles(IndexReader reader, String field, DoubleParser parser)
   at Lucene.Net.Search.FieldCacheImpl.DoubleCache.CreateValue(IndexReader reader, Entry entryKey)
   at Lucene.Net.Search.FieldCacheImpl.Cache.Get(IndexReader reader, Entry key)
   at Lucene.Net.Search.FieldCacheImpl.GetDoubles(IndexReader reader, String field, DoubleParser parser)
   at Lucene.Net.Search.FieldComparator.DoubleComparator.SetNextReader(IndexReader reader, Int32 docBase)
   at Lucene.Net.Search.TopFieldCollector.OneComparatorNonScoringCollector.SetNextReader(IndexReader reader, Int32 docBase)
   at Lucene.Net.Search.IndexSearcher.Search(Weight weight, Filter filter, Collector collector)
   at Lucene.Net.Search.IndexSearcher.Search(Weight weight, Filter filter, Int32 nDocs, Sort sort, Boolean fillFields)
   at Lucene.Net.Search.IndexSearcher.Search(Weight weight, Filter filter, Int32 nDocs, Sort sort)
   at Lucene.Net.Search.Searcher.Search(Query query, Filter filter, Int32 n, Sort sort)
   at Raven.Database.Indexing.Index.IndexQueryOperation.ExecuteQuery(IndexSearcher indexSearcher, Query luceneQuery, Int32 start, Int32 pageSize, IndexQuery indexQuery)
   at Raven.Database.Indexing.Index.IndexQueryOperation.<Query>d__52.MoveNext()
   at Raven.Database.Util.ActiveEnumerable`1..ctor(IEnumerable`1 enumerable)
   at Raven.Database.DocumentDatabase.<>c__DisplayClass91.<Query>b__88(IStorageActionsAccessor actions)
   at Raven.Storage.Esent.TransactionalStorage.ExecuteBatch(Action`1 action, EsentTransactionContext transactionContext)
   at Raven.Storage.Esent.TransactionalStorage.Batch(Action`1 action)
   at Raven.Database.DocumentDatabase.Query(String index, IndexQuery query, Action`1 headerInfo, Action`1 onResult)
   at Raven.Database.DocumentDatabase.Query(String index, IndexQuery query)
   at Raven.Database.Server.Responders.Index.PerformQueryAgainstExistingIndex(IHttpContext context, String index, IndexQuery indexQuery, Etag& indexEtag)
   at Raven.Database.Server.Responders.Index.ExecuteQuery(IHttpContext context, String index, Etag& indexEtag)
   at Raven.Database.Server.Responders.Index.GetIndexQueryResult(IHttpContext context, String index)
   at Raven.Database.Server.Responders.Index.OnGet(IHttpContext context, String index)
   at Raven.Database.Server.Responders.Index.Respond(IHttpContext context)
   at Raven.Database.Server.HttpServer.DispatchRequest(IHttpContext ctx)
   at Raven.Database.Server.HttpServer.HandleActualRequest(IHttpContext ctx)\r\n","ExceptionType":"System.InvalidOperationException","StackTrace":"   at System.Web.Http.ApiController.<InvokeActionWithExceptionFilters>d__1.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Tracing.Tracers.HttpControllerTracer.<ExecuteAsyncCore>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Tracing.ITraceWriterExtensions.<TraceBeginEndAsyncCore>d__21`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__0.MoveNext()","InnerException":{"Message":"An error has occurred.","ExceptionMessage":"The remote server returned an error: (500) Internal Server Error.","ExceptionType":"System.Net.WebException","StackTrace":"   at System.Net.HttpWebRequest.GetResponse()
   at Raven.Client.Connection.HttpJsonRequest.ReadJsonInternal(Func`1 getResponse) in c:\\Builds\\RavenDB-Stable\\Raven.Client.Lightweight\\Connection\\HttpJsonRequest.cs:line 349"

Is this a known issue? How to resolve it without upgrading? (I see that 2750 is the latest stable release) Thanks

Mouhong Lin

unread,
Feb 28, 2014, 12:32:59 AM2/28/14
to rav...@googlegroups.com
I tried to do this with boost:

    public class ItemSimilarityIndex : AbstractIndexCreationTask<ItemSimilarity>
    {
        public ItemSimilarityIndex()
        {
            Map = similarities => from sim in similarities
                                  select new
                                  {
                                      sim.Item1Id,
                                      sim.Item2Id,
                                      sim.Similarity,
                                      sim.Type
                                  }
                                  .Boost((float)(sim.Similarity + 1));

            Index(x => x.Item1Id, Raven.Abstractions.Indexing.FieldIndexing.NotAnalyzed);
            Index(x => x.Item2Id, Raven.Abstractions.Indexing.FieldIndexing.NotAnalyzed);
            Index(x => x.Type, Raven.Abstractions.Indexing.FieldIndexing.NotAnalyzed);
        }
    }

But it seems the result is not exactly sorted by Boost value? I mean:

Query: Type: PageView AND ItemId: 512
Result:

Type: PageView, ItemId: 512, Item2Id: 423, Similarity: 2
Type: PageView, ItemId: 512, Item2Id: 342, Similarity: 1
Type: PageView, ItemId: 512, Item2Id: 445, Similarity: 1.020
Type: PageView, ItemId: 512, Item2Id: 231, Similarity: 1.019
Type: PageView, ItemId: 512, Item2Id: 123, Similarity: 1.021

The result is not exactly sorted by similarity. But it seems the boost do take some effects.
Am I missing something?

Mouhong Lin

unread,
Feb 28, 2014, 12:45:29 AM2/28/14
to rav...@googlegroups.com
I just found that the scores for the last two are the same. But why? Their boost values are not same.


On Friday, February 28, 2014 12:24:45 PM UTC+8, Mouhong Lin wrote:

Oren Eini (Ayende Rahien)

unread,
Feb 28, 2014, 1:05:50 AM2/28/14
to ravendb
Are you running RavenDB in 32 bits mode?

RavenDB Conference

Oren Eini

CEO

Mobile: + 972-52-548-6969

Office:  + 972-4-674-7811

Fax:      + 972-153-4622-7811






--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Mouhong Lin

unread,
Feb 28, 2014, 2:05:15 AM2/28/14
to rav...@googlegroups.com

Yes, the operating system is win server 2008 32bit. i didn't test for outofmemory in local. But i tested boost in local and get the result i mentioned above

You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/rXW1ldwLi4c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

Oren Eini (Ayende Rahien)

unread,
Feb 28, 2014, 3:51:32 AM2/28/14
to ravendb

It is going to be a problem in 32 bits, we probably do not have enough virtual memory left

Reply all
Reply to author
Forward
0 new messages