Query memory usage.

[Ashley Finney]

Hey all,

I'm currently hitting the instance soft memory limit of 256MB by doing this...

q := datastore.NewQuery("MetaPLBStruct").Limit(500)

for t := q.Run(c); ; {

var x MetaLBStruct

key, err := t.Next(&x)

if err == datastore.Done {

break

}

// Do something here

}

Yet I don't run out of memory if I do this...

var lbEnts []MetaLBStruct

lbKeys, err := datastore.NewQuery("MetaPLBStruct").Limit(500).GetAll(c, &lbEnts)

for _, v := range lbEnts {

// Do something here

}

I wrote the routine the first way because I have lots entities to get, much greater then 500 and I wanted to iterate all of them.  I've previously hit a limit of 512(?) entities when using GetAll().

So, what am I doing wrong?  I see python has some kind of batch size that is missing from Go, would this help?

Is there anyway I can see how much memory my instance is using?  I suspect that each .Next() call is taking up quite a bit of memory, perhaps I can early out if I know I'm getting close to the limit?

Many thanks,

Ash.

[Vinny P]

On Wed, Dec 10, 2014 at 6:35 AM, Ashley Finney <2de...@gmail.com> wrote:

I'm currently hitting the instance soft memory limit of 256MB by doing this...

q := datastore.NewQuery("MetaPLBStruct").Limit(500)

So, what am I doing wrong?

This issue has come up before on the google-appengine-go forum; for instance, see this thread: https://groups.google.com/d/msg/google-appengine-go/nB4X5AZlmKQ/K5C2C00NKpoJ . You may find it helpful to cross-post your question there (and there is a discussion on profiling as well). 

With that said, if your intention is to iterate through "much greater than 500" entities you may want to process your entities through a MapReduce approach, or use cursors and only process a batch of entities at one time.

On Wed, Dec 10, 2014 at 6:35 AM, Ashley Finney <2de...@gmail.com> wrote:

I see python has some kind of batch size that is missing from Go, would this help?

 

Batches are available; use the Limit function to declare a batch size, then cursors can move you between one batch to the next.

 

 

-----------------

-Vinny P

Technology & Media Consultant

Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com