Why App engine increase usage memory instantly?

Ky Nguyen

unread,

Oct 19, 2018, 8:23:59 AM10/19/18

to Google App Engine

I'm using GAE flexible to deploy my java servlet application.

And this is my resource config on GAE:

automatic_scaling:

min_num_instances: 1

max_num_instances: 3

cpu_utilization:

target_utilization: 0.7

resources:

cpu: 2

memory_gb: 4

disk_size_gb: 10

After look at to the dashboard of GAE I saw.

My application working fine and the traffic is very low. But I don't know the memory usage dashboard show like image below

I don't have any update on my app, and I have faced one time GAE down with OOM error.

Can anybody explain for me?

Thanks so much.

Carl Smith

unread,

Oct 19, 2018, 10:40:35 AM10/19/18

to Google App Engine

Very odd that this is around the same time that we started to have problems with Cloud SQL queries on AppEngine.

Was there an update to AppEngine around 3rd-8th October that broke a few things and slowed things down?

Our Cloud SQL issue is already logged here:
https://issuetracker.google.com/117895435

Aref Amiri (Cloud Platform Support)

unread,

Oct 23, 2018, 11:47:04 AM10/23/18

to google-a...@googlegroups.com

Hello,

This could be a memory leak since you got an Out Of Memory error or possibly related to an update on App Engine. There are many possibilities that could lead to memory leak. All the causes mainly are related to the application itself. These include and not limited to resource memory allocation, as they might not be allocated properly in the application. In addition, I noticed some spikes on the graph provided that could possibly be related to the workload. As the workload increases per instance, this could lead to a memory leak. I would suggest to post the issue on Stackoverflow and the community can definitely be a great help and propose effective solutions.There are some examples that you can refer to as they are similar cases to yours:

1: https://stackoverflow.com/questions/46359298/how-to-fix-memory-leak-in-my-application

2: https://stackoverflow.com/questions/40119/how-to-find-a-java-memory-leak?rq=1

You can also use the Google Stackdriver Debugger which could be very useful in this case.

After checking the solutions, if you believe the issue is related to App Engine, you could post the issue on Google Issue Tracker, which would allow us to investigate the issue further and if necessary, route it to a specialized team.

Carl, I have double checked here on my end and could not find any troublesome updates released on the specified dates. To get information about current and past outages you can use the Google Cloud Status Dashboard page. Regarding your issue, it has been routed to the specialized team and you will be notified once there is an update.

Ky Nguyen

unread,

Oct 24, 2018, 7:19:33 AM10/24/18

to google-a...@googlegroups.com

I sure the problem not come from application.

because it had been deployed it for long time ago and it just use memory around 800MB until Oct 07.

After OOM occurred, we just change config on app.yaml to increase from memory_gb: 1 to memory_gb: 4.

After add allocated memory on GAE I saw memory usage increase steadily.

Do you have any idea to update?

On Tue, Oct 23, 2018 at 10:47 PM 'Aref Amiri (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com> wrote:

Hello,

This could be a memory leak since you got an Out Of Memory error or possibly related to an update on App Engine. There are many possibilities that could lead to memory leak. All the causes mainly are related to the application itself. These include and not limited to resource memory allocation, as they might not be allocated properly in the application. In addition, I noticed some spikes on the graph provided that could possibly be related to the workload. As the workload increases, it creates more instances and hence causing memory leak. I would suggest to post the issue on Stackoverflow and the community can definitely be a great help and propose effective solutions.There are some examples that you can refer to as they are similar cases to yours:

1: https://stackoverflow.com/questions/46359298/how-to-fix-memory-leak-in-my-application
2: https://stackoverflow.com/questions/40119/how-to-find-a-java-memory-leak?rq=1

You can also use the Google Stackdriver Debugger which could be very useful in this case.

After checking the solutions, if you believe the issue is related to App Engine, you could post the issue on Google Issue Tracker, which would allow us to investigate the issue further and if necessary, route it to a specialized team.

Carl, I have double checked here on my end and could not find any troublesome updates released on the specified dates. To get information about current and past outages you can use the Google Cloud Status Dashboard page. Regarding your issue, it has been routed to the specialized team and you will be notified once there is an update.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/5a8c3e3f-029f-4d44-811a-7052dfd3389f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Nguyễn Lương Kỳ

-------------------------------------------------------------------------

Phone: 0975 072 768
Skype: luongky06

Email: luon...@gmail.com

Yahoo: luo ngk...@yahoo.com.vn

-------------------------------------------------------------------------

Amit (Google Cloud Support)

unread,

Oct 25, 2018, 11:18:56 AM10/25/18

to Google App Engine

Hello Nguyễn

High memory utilization could happen due to your application handling more traffic than before or higher workloads than before. If you believe your issue is not related to any of this and nothing has changed in your code yet, then I would suggest to open a private issue in issue tracker and share all the details of you issue including your project ID, exact timeframe when you observed the issue and any changes you made in your configuration so that we can look into your issue and understand its scope.

On Wednesday, October 24, 2018 at 7:19:33 AM UTC-4, Ky Nguyen wrote:

I sure the problem not come from application.
because it had been deployed it for long time ago and it just use memory around 800MB until Oct 07.
After OOM occurred, we just change config on app.yaml to increase from memory_gb: 1 to memory_gb: 4.

After add allocated memory on GAE I saw memory usage increase steadily.
Do you have any idea to update?

On Tue, Oct 23, 2018 at 10:47 PM 'Aref Amiri (Cloud Platform Support)' via Google App Engine <google-appengine@googlegroups.com> wrote:

Hello,

This could be a memory leak since you got an Out Of Memory error or possibly related to an update on App Engine. There are many possibilities that could lead to memory leak. All the causes mainly are related to the application itself. These include and not limited to resource memory allocation, as they might not be allocated properly in the application. In addition, I noticed some spikes on the graph provided that could possibly be related to the workload. As the workload increases, it creates more instances and hence causing memory leak. I would suggest to post the issue on Stackoverflow and the community can definitely be a great help and propose effective solutions.There are some examples that you can refer to as they are similar cases to yours:

1: https://stackoverflow.com/questions/46359298/how-to-fix-memory-leak-in-my-application
2: https://stackoverflow.com/questions/40119/how-to-find-a-java-memory-leak?rq=1

You can also use the Google Stackdriver Debugger which could be very useful in this case.

After checking the solutions, if you believe the issue is related to App Engine, you could post the issue on Google Issue Tracker, which would allow us to investigate the issue further and if necessary, route it to a specialized team.

Carl, I have double checked here on my end and could not find any troublesome updates released on the specified dates. To get information about current and past outages you can use the Google Cloud Status Dashboard page. Regarding your issue, it has been routed to the specialized team and you will be notified once there is an update.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/5a8c3e3f-029f-4d44-811a-7052dfd3389f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Ky Nguyen

unread,

Oct 26, 2018, 1:30:47 AM10/26/18

to google-a...@googlegroups.com

Hi Amit,

After investigated on my application I found out the reason make memory increase.

We had enable google monitoring on Stackdriver with Uptime check. so uptime check request from stack driver will call to our endpoint every 5 minute.

It make application high traffic than before, and the memory also increase steadily on every stack diver run uptime check service.

On the endpoint that Uptime check will call, We just implement rest client (by spring Resttemplate) to call 3rd party service.

But look like the memory not release on each time endpoint called. I run application on my local with jetty embedded server but memory not increase too much like on GAE.

Do you have any idea to help.

Thanks.

On Thu, Oct 25, 2018 at 10:19 PM 'Amit (Google Cloud Support)' via Google App Engine <google-a...@googlegroups.com> wrote:

Hello Nguyễn

High memory utilization could happen due to your application handling more traffic than before or higher workloads than before. If you believe your issue is not related to any of this and nothing has changed in your code yet, then I would suggest to open a private issue in issue tracker and share all the details of you issue including your project ID, exact timeframe when you observed the issue and any changes you made in your configuration so that we can look into your issue and understand its scope.

On Wednesday, October 24, 2018 at 7:19:33 AM UTC-4, Ky Nguyen wrote:

I sure the problem not come from application.
because it had been deployed it for long time ago and it just use memory around 800MB until Oct 07.
After OOM occurred, we just change config on app.yaml to increase from memory_gb: 1 to memory_gb: 4.

After add allocated memory on GAE I saw memory usage increase steadily.
Do you have any idea to update?

On Tue, Oct 23, 2018 at 10:47 PM 'Aref Amiri (Cloud Platform Support)' via Google App Engine <google-a...@googlegroups.com> wrote:

Hello,

This could be a memory leak since you got an Out Of Memory error or possibly related to an update on App Engine. There are many possibilities that could lead to memory leak. All the causes mainly are related to the application itself. These include and not limited to resource memory allocation, as they might not be allocated properly in the application. In addition, I noticed some spikes on the graph provided that could possibly be related to the workload. As the workload increases, it creates more instances and hence causing memory leak. I would suggest to post the issue on Stackoverflow and the community can definitely be a great help and propose effective solutions.There are some examples that you can refer to as they are similar cases to yours:

1: https://stackoverflow.com/questions/46359298/how-to-fix-memory-leak-in-my-application
2: https://stackoverflow.com/questions/40119/how-to-find-a-java-memory-leak?rq=1

You can also use the Google Stackdriver Debugger which could be very useful in this case.

After checking the solutions, if you believe the issue is related to App Engine, you could post the issue on Google Issue Tracker, which would allow us to investigate the issue further and if necessary, route it to a specialized team.

Carl, I have double checked here on my end and could not find any troublesome updates released on the specified dates. To get information about current and past outages you can use the Google Cloud Status Dashboard page. Regarding your issue, it has been routed to the specialized team and you will be notified once there is an update.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/5a8c3e3f-029f-4d44-811a-7052dfd3389f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Nguyễn Lương Kỳ
-------------------------------------------------------------------------
           Phone: 0975 072 768
Skype: luongky06
Email:   luon...@gmail.com
Yahoo: luo ngk...@yahoo.com.vn
-------------------------------------------------------------------------

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/bd88cdc4-cb49-4f0c-a16c-501c5a4d4f4a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Amit (Google Cloud Support)

unread,

Oct 26, 2018, 5:41:03 PM10/26/18

to Google App Engine

Hello Nguyen,

From your given description, as your application code does not properly release memory, you need to find out why it is not releasing the memory when needed. To do that I would recommend to use this Java Stackdriver Profiler, consists of the profiling agent, which collects the data, and a console interface on Google Cloud Platform, which lets you view and analyze the data collected by the agent. Profiling heap(memory) consumption helps you find potential inefficiencies and memory leaks in your programs.

If you see from the Profiler Interface, your application’s memory usage doesn’t go up with your instance’s memory usage, then this would be a sign of bug which should report in issue tracker as discussed earlier. Otherwise if only the application memory goes up, then you need to fix the code.

On Friday, October 26, 2018 at 1:30:47 AM UTC-4, Ky Nguyen wrote:

Hi Amit,

After investigated on my application I found out the reason make memory increase.
We had enable google monitoring on Stackdriver with Uptime check. so uptime check request from stack driver will call to our endpoint every 5 minute.
It make application high traffic than before, and the memory also increase steadily on every stack diver run uptime check service.
On the endpoint that Uptime check will call, We just implement rest client (by spring Resttemplate) to call 3rd party service.
But look like the memory not release on each time endpoint called. I run application on my local with jetty embedded server but memory not increase too much like on GAE.

Do you have any idea to help.

Thanks.

On Thu, Oct 25, 2018 at 10:19 PM 'Amit (Google Cloud Support)' via Google App Engine <google-appengine@googlegroups.com> wrote:

Hello Nguyễn

High memory utilization could happen due to your application handling more traffic than before or higher workloads than before. If you believe your issue is not related to any of this and nothing has changed in your code yet, then I would suggest to open a private issue in issue tracker and share all the details of you issue including your project ID, exact timeframe when you observed the issue and any changes you made in your configuration so that we can look into your issue and understand its scope.

On Wednesday, October 24, 2018 at 7:19:33 AM UTC-4, Ky Nguyen wrote:

I sure the problem not come from application.
because it had been deployed it for long time ago and it just use memory around 800MB until Oct 07.
After OOM occurred, we just change config on app.yaml to increase from memory_gb: 1 to memory_gb: 4.

After add allocated memory on GAE I saw memory usage increase steadily.
Do you have any idea to update?

On Tue, Oct 23, 2018 at 10:47 PM 'Aref Amiri (Cloud Platform Support)' via Google App Engine <google-appengine@googlegroups.com> wrote:

Hello,

This could be a memory leak since you got an Out Of Memory error or possibly related to an update on App Engine. There are many possibilities that could lead to memory leak. All the causes mainly are related to the application itself. These include and not limited to resource memory allocation, as they might not be allocated properly in the application. In addition, I noticed some spikes on the graph provided that could possibly be related to the workload. As the workload increases, it creates more instances and hence causing memory leak. I would suggest to post the issue on Stackoverflow and the community can definitely be a great help and propose effective solutions.There are some examples that you can refer to as they are similar cases to yours:

1: https://stackoverflow.com/questions/46359298/how-to-fix-memory-leak-in-my-application
2: https://stackoverflow.com/questions/40119/how-to-find-a-java-memory-leak?rq=1

You can also use the Google Stackdriver Debugger which could be very useful in this case.

After checking the solutions, if you believe the issue is related to App Engine, you could post the issue on Google Issue Tracker, which would allow us to investigate the issue further and if necessary, route it to a specialized team.

Carl, I have double checked here on my end and could not find any troublesome updates released on the specified dates. To get information about current and past outages you can use the Google Cloud Status Dashboard page. Regarding your issue, it has been routed to the specialized team and you will be notified once there is an update.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/5a8c3e3f-029f-4d44-811a-7052dfd3389f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Nguyễn Lương Kỳ
-------------------------------------------------------------------------
           Phone: 0975 072 768
Skype: luongky06
Email:   luon...@gmail.com
Yahoo: luo ngk...@yahoo.com.vn
-------------------------------------------------------------------------

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/bd88cdc4-cb49-4f0c-a16c-501c5a4d4f4a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Ky Nguyen

unread,

Oct 30, 2018, 1:11:31 AM10/30/18

to google-a...@googlegroups.com

Hi Amit,

Thanks for your support, I had fixed my api endpoint to avoid memory leak.

But look like it is not a root cause.

I deploy new application with the same config and let it run a few day with no traffic, but memory still increase.

I'm confusing if the problem come from cloud sql connection.

I'm using Hikaricp to connect to cloud sql (Postgresql) in my web application and I saw in monitor dashboard of memory usage still increase.

I saw in Stackdriver log that application connect to cloud sql frequently.

I don't have any idea to fix it, can you suggest me a way to connect to cloud sql via Hikaricp ?

Thanks

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/5a8c3e3f-029f-4d44-811a-7052dfd3389f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Nguyễn Lương Kỳ
-------------------------------------------------------------------------
           Phone: 0975 072 768
Skype: luongky06
Email:   luon...@gmail.com
Yahoo: luo ngk...@yahoo.com.vn
-------------------------------------------------------------------------

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/bd88cdc4-cb49-4f0c-a16c-501c5a4d4f4a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Nguyễn Lương Kỳ
-------------------------------------------------------------------------
           Phone: 0975 072 768
Skype: luongky06
Email:   luon...@gmail.com
Yahoo: luo ngk...@yahoo.com.vn
-------------------------------------------------------------------------

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/d542b586-c4e4-4449-82be-47a0f7017baa%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Amit (Google Cloud Support)

unread,

Oct 31, 2018, 11:41:20 AM10/31/18

to Google App Engine

Hello Nguyen,

By doing some internal research, I didn't find any current bugs related to memory leak in App Engine runtime. The code you are using for Cloud SQL connection could have some issue and causing this high memory utilization. I believe posting in Stack Overflow using ‘google-cloud-platform’ and ‘google-app-engine’ tag with the snippet of your Cloud SQL connection code including screenshots of your stackdriver logs could help you get more information regarding this error.

On Tuesday, October 30, 2018 at 1:11:31 AM UTC-4, Ky Nguyen wrote:

Hi Amit,

Thanks for your support, I had fixed my api endpoint to avoid memory leak.
But look like it is not a root cause.
I deploy new application with the same config and let it run a few day with no traffic, but memory still increase.

I'm confusing if the problem come from cloud sql connection.
I'm using Hikaricp to connect to cloud sql (Postgresql) in my web application and I saw in monitor dashboard of memory usage still increase.

I saw in Stackdriver log that application connect to cloud sql frequently.

I don't have any idea to fix it, can you suggest me a way to connect to cloud sql via Hikaricp ?
Thanks

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/5a8c3e3f-029f-4d44-811a-7052dfd3389f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Nguyễn Lương Kỳ
-------------------------------------------------------------------------
           Phone: 0975 072 768
Skype: luongky06
Email:   luon...@gmail.com
Yahoo: luo ngk...@yahoo.com.vn
-------------------------------------------------------------------------

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/bd88cdc4-cb49-4f0c-a16c-501c5a4d4f4a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Nguyễn Lương Kỳ
-------------------------------------------------------------------------
           Phone: 0975 072 768
Skype: luongky06
Email:   luon...@gmail.com
Yahoo: luo ngk...@yahoo.com.vn
-------------------------------------------------------------------------

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/d542b586-c4e4-4449-82be-47a0f7017baa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply all

Reply to author

Forward