Re: [RavenDB] Sparrow FastList and BittableJsonReaderObject stays in memory after query?

198 views
Skip to first unread message

Oren Eini (Ayende Rahien)

unread,
Aug 2, 2018, 6:40:07 AM8/2/18
to ravendb
The most likely reason is that this is the cache, holding them in memory. We are caching requests to be able to provide you with faster responses. 
Check the path of the references for this, you'll likely find it goes to the cache

Hibernating Rhinos Ltd  

Oren Eini l CEO Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 


On Thu, Aug 2, 2018 at 12:25 AM, Victor Potatis Maximus <victorno...@gmail.com> wrote:
Hello, i'm trying to query a list of documents, which only has one long property.
The list can be anywhere from 500,000 to 10,000,000 documents in size, but in this example there's 524,602 documents.
After my query has completed i'm stuck with a bunch of sparrow and ravendb objects, which never seems to be collected by gc or something.
They're quite substantial in size compared to the actual list i queried, so it's kind of a big issue for me application.
Please have a look at the attached image of a memory snapshot.


Any insights or advice would be appreciated.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Victor Potatis Maximus

unread,
Aug 2, 2018, 7:51:43 AM8/2/18
to RavenDB - 2nd generation document database
Okay, but please tell me how i could do this differently then.
I'm sure there's a way to store a list of 700,000 strings, without using way over 1 gb of memory.
I'm simply making the query and selecting to list of strings. (It's stored as long in the Value property)
= new List<string>((await rdb.GetSpecialDocumentsAsync()).Select(sd => sd.Value.ToString()));

My application is using less than 400MB before the query and around 1900MB after the query.
Speed is not my main concern here, so please tell me if there's a solution.
Thanks
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

Victor Potatis Maximus

unread,
Aug 2, 2018, 7:53:18 AM8/2/18
to RavenDB - 2nd generation document database
Oh and this request is only made once in the lifetime of the application.
Would it be possible to disable the request caching, if that's the issue?

Oren Eini (Ayende Rahien)

unread,
Aug 2, 2018, 7:57:04 AM8/2/18
to ravendb
      var asyncQuery = asyncSession.Query<Item>().Customize(x => x.NoCaching().WaitForNonStaleResults()).ToListAsync();

             
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

Victor Potatis Maximus

unread,
Aug 2, 2018, 8:28:53 AM8/2/18
to RavenDB - 2nd generation document database
Since NoCaching didn't make a difference for me, i tried NoTracking as well.
No difference at all, unfortunately. Please see attached images

using (var session = store.OpenAsyncSession())
            {
                    specialDocs = await session.Query<SpecialDocument>()
                        .Customize(x => x.NoTracking().NoCaching().WaitForNonStaleResults())
                        .ToListAsync();
                
            }

Am i doing something completely wrong?....
Query_with_notracking_nocaching.jpg
Query_without_customization.jpg

Oren Eini (Ayende Rahien)

unread,
Aug 2, 2018, 8:30:02 AM8/2/18
to ravendb
Please show the full tracking of this memory. Who is holding it?
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

Victor Potatis Maximus

unread,
Aug 2, 2018, 1:29:01 PM8/2/18
to RavenDB - 2nd generation document database
I hope this is of any use.
Seems like it's coming from GlobalCache, but i've executed DisableAggressiveCaching() on the store.
And the query had both NoTracking and NoCaching.
For some reason the memory usage drops by 50%, as soon as i take a snapshot (as you can see on the first image).

Thanks
InkedScreenshot_161_LI.jpg
Screenshot_162.png

Victor Potatis Maximus

unread,
Aug 2, 2018, 1:35:17 PM8/2/18
to RavenDB - 2nd generation document database
Thanks

torsdag den 2. august 2018 kl. 12.40.07 UTC+2 skrev Oren Eini:
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

Victor Potatis Maximus

unread,
Aug 2, 2018, 1:52:51 PM8/2/18
to RavenDB - 2nd generation document database
I've now set
store.Conventions.MaxHttpCacheSize = new Size(0, 0);
before initializing the store.
Everything is exactly the same.

torsdag den 2. august 2018 kl. 12.40.07 UTC+2 skrev Oren Eini:
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

Oren Eini (Ayende Rahien)

unread,
Aug 2, 2018, 3:33:07 PM8/2/18
to ravendb
Not sure that I'm following. Does this work for you now?
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

Victor Potatis Maximus

unread,
Aug 2, 2018, 4:25:21 PM8/2/18
to RavenDB - 2nd generation document database
Sorry if i wasn't clear. No i've tried everything and i can't query a big list from the database,
without making my application memory usage explode.

I feel like i've tried everything-

Victor Potatis Maximus

unread,
Aug 2, 2018, 4:26:45 PM8/2/18
to RavenDB - 2nd generation document database
Everything seems to stay exactly the same, in terms of memory usage.
None of the things i've tried have made any observable difference it seems.

Oren Eini (Ayende Rahien)

unread,
Aug 2, 2018, 4:29:17 PM8/2/18
to ravendb
Can you send us a repro?
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
Message has been deleted

Victor Potatis Maximus

unread,
Aug 2, 2018, 4:46:45 PM8/2/18
to RavenDB - 2nd generation document database
I can't, but i don't think that would be necessary either.
I'll include all of the code in question here though:

public class DatabaseTest
    {       

        public DatabaseTest()
        {
            var Store = new DocumentStore
            {
                Urls = new[] { "http://localhost:3113" },
                Database = "TestDB"
            };
            
            Store.Conventions.MaxHttpCacheSize = new Size(0, 0);

            Store.Initialize();
            Task.Run(() => Test(Store));
        }

        public async Test(IDocumentStore store)
        {
            Raven rdb = new Raven(store)
            var docs = await rdb.GetSimpleDocumentsAsync()
             //From here on, memory has exploded
        }
    }

public class Raven
    {
        readonly IDocumentStore CurrentStore;

        public Raven(IDocumentStore store)
        {
            CurrentStore = store;
        }

        public async Task<List<SimpleDocument>> GetSimpleDocumentsAsync()
        {
            List<SimpleDocument> simpleDocs;

            using (var session = CurrentStore.OpenAsyncSession())
            {
                    simpleDocs = await session.Query<SpecialDocument>()
                        .Customize(x => x.NoTracking().NoCaching().WaitForNonStaleResults())
                        .ToListAsync();
                
            }

            return  simpleDocs;
        }

     }

I'm pretty sure everything is right, according to your documentation.

Victor Potatis Maximus

unread,
Aug 2, 2018, 4:50:35 PM8/2/18
to RavenDB - 2nd generation document database
Sorry, i didn't include the SpecialDocument:

public class SpecialDocument
    {
        public string Value { get; set; }

Oren Eini (Ayende Rahien)

unread,
Aug 2, 2018, 5:02:35 PM8/2/18
to ravendb
What is the size of the string there?
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

Victor Potatis Maximus

unread,
Aug 2, 2018, 5:38:15 PM8/2/18
to RavenDB - 2nd generation document database
Only 10 to 12 chars

Victor Potatis Maximus

unread,
Aug 3, 2018, 10:53:03 AM8/3/18
to RavenDB - 2nd generation document database
I've created a clean project to test and i'm definitely still having the same issue.
I'm wondering if it's like this for everyone?
I just can't do all of my comparisons on the database end, so i have to query all documents in the collection at software start.
It really shouldn't be this hard to query a large list of extremely reasonable size documents, without incurring a 1.5gb+ cache for just 800,000 documents.
We're using ravendb as a local server and at this point i'm considering just writing the strings to a file... Though that wouldn't be nearly as convenient for us.

Victor Potatis Maximus

unread,
Aug 3, 2018, 10:54:55 AM8/3/18
to RavenDB - 2nd generation document database
Could i do the query manually with a web request or something, to try to avoid what seems like a cached request?
I mean without using the ravendb client?

Maxim Buryak

unread,
Aug 5, 2018, 8:35:45 AM8/5/18
to rav...@googlegroups.com
Hi, 
Please see my other response about work around for this issue.
About the web request: first, I can't recommend you to go this path, unless you want to bypass all the caching and deserialization mechanisms. The 4.0 http api is not documented yet, so the best way to go will be to use fiddler (see details about fiddler usage here: https://ravendb.net/docs/article-page/4.0/Csharp/server/troubleshooting/sending-support-ticket) to discover the needed request.

Oren Eini (Ayende Rahien)

unread,
Aug 5, 2018, 11:26:23 AM8/5/18
to ravendb
To reiterate what Maxim said, please use streaming, instead of doing things manually
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

Victor Potatis Maximus

unread,
Aug 5, 2018, 3:37:36 PM8/5/18
to RavenDB - 2nd generation document database
Could you send me a link to your response?
I'm guessing it wasn't in this thread?

Thanks, would be a life saver

Oren Eini (Ayende Rahien)

unread,
Aug 7, 2018, 4:19:22 AM8/7/18
to ravendb
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages