Re: [RavenDB] Sparrow FastList and BittableJsonReaderObject stays in memory after query?

[Oren Eini (Ayende Rahien)]

The most likely reason is that this is the cache, holding them in memory. We are caching requests to be able to provide you with faster responses. 

Check the path of the references for this, you'll likely find it goes to the cache

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

On Thu, Aug 2, 2018 at 12:25 AM, Victor Potatis Maximus <victorno...@gmail.com> wrote:

Hello, i'm trying to query a list of documents, which only has one long property.
The list can be anywhere from 500,000 to 10,000,000 documents in size, but in this example there's 524,602 documents.

After my query has completed i'm stuck with a bunch of sparrow and ravendb objects, which never seems to be collected by gc or something.

They're quite substantial in size compared to the actual list i queried, so it's kind of a big issue for me application.
Please have a look at the attached image of a memory snapshot.


Any insights or advice would be appreciated.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Victor Potatis Maximus]

Okay, but please tell me how i could do this differently then.
I'm sure there's a way to store a list of 700,000 strings, without using way over 1 gb of memory.
I'm simply making the query and selecting to list of strings. (It's stored as long in the Value property)

= new List<string>((await rdb.GetSpecialDocumentsAsync()).Select(sd => sd.Value.ToString()));

My application is using less than 400MB before the query and around 1900MB after the query.
Speed is not my main concern here, so please tell me if there's a solution.

Thanks

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Victor Potatis Maximus]

Oh and this request is only made once in the lifetime of the application.
Would it be possible to disable the request caching, if that's the issue?

[Oren Eini (Ayende Rahien)]

      var asyncQuery = asyncSession.Query<Item>().Customize(x => x.NoCaching().WaitForNonStaleResults()).ToListAsync();

             

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Victor Potatis Maximus]

Since NoCaching didn't make a difference for me, i tried NoTracking as well.
No difference at all, unfortunately. Please see attached images

using (var session = store.OpenAsyncSession())

            {

                    specialDocs = await session.Query<SpecialDocument>()

                        .Customize(x => x.NoTracking().NoCaching().WaitForNonStaleResults())

                        .ToListAsync();

                

            }

Am i doing something completely wrong?....

[Oren Eini (Ayende Rahien)]

Please show the full tracking of this memory. Who is holding it?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Victor Potatis Maximus]

I hope this is of any use.
Seems like it's coming from GlobalCache, but i've executed DisableAggressiveCaching() on the store.

And the query had both NoTracking and NoCaching.
For some reason the memory usage drops by 50%, as soon as i take a snapshot (as you can see on the first image).

Thanks

[Victor Potatis Maximus]

Could i somehow set maxhttpcachesize?
https://ravendb.net/docs/article-page/4.0/csharp/client-api/configuration/conventions#maxhttpcachesize
Where do i use that?

Thanks

torsdag den 2. august 2018 kl. 12.40.07 UTC+2 skrev Oren Eini:

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Victor Potatis Maximus]

I've now set

store.Conventions.MaxHttpCacheSize = new Size(0, 0);

before initializing the store.
Everything is exactly the same.

torsdag den 2. august 2018 kl. 12.40.07 UTC+2 skrev Oren Eini:

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

Not sure that I'm following. Does this work for you now?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Victor Potatis Maximus]

Sorry if i wasn't clear. No i've tried everything and i can't query a big list from the database,
without making my application memory usage explode.

I feel like i've tried everything-

[Victor Potatis Maximus]

Everything seems to stay exactly the same, in terms of memory usage.
None of the things i've tried have made any observable difference it seems.

[Oren Eini (Ayende Rahien)]

Can you send us a repro?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Victor Potatis Maximus]

I can't, but i don't think that would be necessary either.
I'll include all of the code in question here though:

public class DatabaseTest

    {       

        public DatabaseTest()

        {

            var Store = new DocumentStore

            {

                Urls = new[] { "http://localhost:3113" },

                Database = "TestDB"

            };

            

            Store.Conventions.MaxHttpCacheSize = new Size(0, 0);

            Store.Initialize();

            Task.Run(() => Test(Store));

        }

        public async Test(IDocumentStore store)

        {

            Raven rdb = new Raven(store)

            var docs = await rdb.GetSimpleDocumentsAsync()

             //From here on, memory has exploded

        }

    }

public class Raven

    {

        readonly IDocumentStore CurrentStore;

        public Raven(IDocumentStore store)

        {

            CurrentStore = store;

        }

        public async Task<List<SimpleDocument>> GetSimpleDocumentsAsync()

        {

            List<SimpleDocument> simpleDocs;

            using (var session = CurrentStore.OpenAsyncSession())

            {

                    simpleDocs = await session.Query<SpecialDocument>()

                        .Customize(x => x.NoTracking().NoCaching().WaitForNonStaleResults())

                        .ToListAsync();

                

            }

            return  simpleDocs;

        }

     }

I'm pretty sure everything is right, according to your documentation.

[Victor Potatis Maximus]

Sorry, i didn't include the SpecialDocument:

public class SpecialDocument

    {

        public string Value { get; set; }

[Oren Eini (Ayende Rahien)]

What is the size of the string there?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Victor Potatis Maximus]

Only 10 to 12 chars

[Victor Potatis Maximus]

I've created a clean project to test and i'm definitely still having the same issue.
I'm wondering if it's like this for everyone?
I just can't do all of my comparisons on the database end, so i have to query all documents in the collection at software start.
It really shouldn't be this hard to query a large list of extremely reasonable size documents, without incurring a 1.5gb+ cache for just 800,000 documents.

We're using ravendb as a local server and at this point i'm considering just writing the strings to a file... Though that wouldn't be nearly as convenient for us.

[Victor Potatis Maximus]

Could i do the query manually with a web request or something, to try to avoid what seems like a cached request?
I mean without using the ravendb client?

[Maxim Buryak]

Hi, 

Please see my other response about work around for this issue.

About the web request: first, I can't recommend you to go this path, unless you want to bypass all the caching and deserialization mechanisms. The 4.0 http api is not documented yet, so the best way to go will be to use fiddler (see details about fiddler usage here: https://ravendb.net/docs/article-page/4.0/Csharp/server/troubleshooting/sending-support-ticket) to discover the needed request.

[Oren Eini (Ayende Rahien)]

To reiterate what Maxim said, please use streaming, instead of doing things manually

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Victor Potatis Maximus]

Could you send me a link to your response?
I'm guessing it wasn't in this thread?

Thanks, would be a life saver

[Oren Eini (Ayende Rahien)]

https://groups.google.com/d/msg/ravendb/uoyucdU69Ms/MHUsQGKSBwAJ

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.