Exceeded Soft Private memory usage of 128MB

[Richard Cheesmar]

Hi, all

I had some errors on a POST handler today, which by the way fully completed, it just raised exceptions...

An example exception is: Exceeded soft private memory limit of 128 MB with 133 MB after servicing 313 requests total.

I'm still in the middle of checking the source for problems, however this is the first time this has occurred, so I figure I might just need to upgrade the instances which brings me to my questions:


1. How do you do that with vanilla App Engine, i.e. no VM setup
2. Can you upgrade instances at certain times, so schedule upgrades on a hour by hour basis?


Thanks

[Richard Cheesmar]

Any chance of a response Google or do I have to put questions like this on Stackoverflow ???

[John Smith]

you can take a look at the App engine Modules page

if you want additional memory on classic app engine you could upgrade your instances to B2/F2 instance class

you have to add this to your app.yaml file:

instance_class: F2

please note that F2 instance costs twice as F1 instance would.

[Richard Cheesmar]

Thanks, John, I'll take a look.

On Thursday, January 14, 2016 at 9:43:04 PM UTC+2, Richard Cheesmar wrote:

[Nick (Cloud Platform Support)]

John Smith has the correct answer. The instance class, which determines the amount of memory available, is selected at deployment time. The link to the documentation he provided is also the correct page. Best of luck in debugging your memory use!

[Kaan Soral]

I have an app that kills instances after just 30 requests sometimes, it functions perfectly tho, the instances die pretty gracefully, my assumption is that they don't take anything with them, rather handle all their requests and die after that

I don't upgrade to F2 as it increases the costs significantly, it's harder to utilise F2 instances efficiently

My .02

[Christian F. Howes]

FYI, if you use NDB there is a memory leak:  https://code.google.com/p/googleappengine/issues/detail?id=9610  the google folks are having trouble pinning it down, but it's there.  i just let my instances restart as they run out of memory as i have not been able to find my other memory leaks (i'm sure i have more then just the NDB leak).

[Kaan Soral]

Interesting, thanks for the link

The app I mentioned used db, my new app uses ndb tho I don't experience memory issues frequently with it

The app that experiences memory leaks has crons that pulls and processes 40 entities at a time, entities are <1MB, so when instances are flooded with these crons, memory issues occur frequently

Interestingly, if the routine is slow, they occur less, but if the routine is fast, it occurs every 30-40 requests

[Nick (Cloud Platform Support)]

Hey Kaan,

Interested to try to analyze the source of the issue - what do you mean by "if the routine is slow" - does this refer to the cron schedule being a larger interval, or the time for requests to execute? Are you able to run memory diagnostics in-request to log the rate at which memory gets used-up?

[Kaan Soral]

Scenario A

1) Fetch 40 entities

2) Fetch an external url, takes 1-15 seconds

3) Put 40 entities async

Scenario B

1) Fetch 40 entities

2) Internal computation, no delay

3) Put 40 entities async

So the issue is seldom at A, yet it floods the logs at B

At both scenarios, multitude of these operations are ran, 40M entities for B, <<1M for A

(A is an hourly operation, while B is more of a one-time thing)

Since there is no observable data/operation loss, I didn't debug the issue

[Nick (Cloud Platform Support)]

Hmm, it's hard to tell what might cause the issue from this level of abstraction, but if you ever felt like it, you could produce a small reproduction and a technical report and pass it off to the Public Issue Tracker. It could be a complex case of WorkAsIntended, or it could be exposing a little-realized issue, the only way is to investigate deeper!