Python 3.7 service killed after exceeding memory limit

[vvv vvv]

This is my cron job, when I run it locally on my intel multi-core i7 it uses less than 200 MB, however on the cloud it uses more than 500 MB and thus gets killed. It can never terminate. Is this because it only runs on one core on the cloud? What this service does is spawn a few threads and then download pages from lots of feeds. Any idea on how to get it to use less memory?

[George (Cloud Platform Support)]

Is your Cron handler deployed in the Standard or Flexible Environment? You seem to use a B4 instance class. To determine whether the instance class is indeed at the origin of these errors, you may consider upgrading to a higher instance class, and see if this makes any difference, and if then the Cron handler runs as expected. 

Instance class is not the only parameter that comes into consideration. You handler may generate memory leaks, and you should verify your handler's code accordingly. The fact that your handler spawns a few threads, and downloads pages from various feeds, is in itself a variable and to some extent random parameter, as the handler's workload and required memory may vary unpredictably. Is there any difference in behavior if you limit the number of feeds and pages to download to a fixed total maximum? If you don't notice any difference at first, you may try again with a lower number of feeds, to finally determine if this parameter is a factor, and to what extent. 

This discussion group is oriented more towards general opinions, trends, and issues of general nature touching the app engine. For coding and programming architecture, as well as tuning Cron handlers, you may be better served in dedicated forums such as stackoverflow, where experienced programmers are within reach and ready to help. 

[vvv vvv]

@George I understand that this mailing list is not meant to be used for architecture issues, but when my software uses around 170 MB of RAM on my laptop but gets killed by the AppEngine on the cloud because it keeps running out of memory, isn't this a potential issue with the platform? I haven't tried setting a limit of 1 GB of RAM in the cloud, as the problem for me is that it keeps using up all its memory (i.e it will eventually hit the 1GB limit if I add more feeds). Using fewer threads is something that does help indeed, but still it doesn't fix the problem. 

[vvv vvv]

BTW this is in the Standard environment 

[Rahul Ravindran]

Hello,

  Your measurement of your application on your laptop does not accurately represent all the memory used. Firstly, you will need to look at the RSS memory for the process. In addition, any resources taken by the operating system, kernel are not accounted for in your measurement but is accounted in ours. We are not running a Python process alone, but giving you a complete, isolated linux runtime environment in addition to the python runtime environment.

~Rahul.

On Tue, Nov 20, 2018 at 2:17 PM vvv vvv <vak...@gmail.com> wrote:

BTW this is in the Standard environment 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/12f271c6-4eb9-4fea-a022-2bb581471e14%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.